Kodak Color Control Patches Kodak Gray

Kodak Color Control Patches 0 - - _,...,...,. Blue Green Yellow Red White 3/Color . Black

Kodak Gray Scale

A 1 2 a 4 s ------

A Study of Compile-time Metaobject Protocol

:J/1\'-(Jt.-S~..)(:$/;:t::;i':_;·I'i f-.· 7 •[:]f-. :::JJl. (:~9 ~~Jf'?E

Shigeru Chiba chiba~is.s.u-tokyo.ac.jp

November 1996

A Dissertation Submitted to Department of Information Science Graduate School of Science The Un iversity of Tokyo

In Partial Fulfillment of the Requirements For the Degree of Doctor of Science.

A Metaobject Protocol for Enabling Better C++ Libraries

Shigeru Chiba

Department of Information Science The University of Tokyo chiba~is.s.u-tokyo.ac.jp Copyrighl @ 1996 by Shi geru Chiba. All Ri ghts Reserved. iii iv

Abstract

C++ cannot be used to implement control/data abstractions as a library if their implementations require specialized code for each user code. This problem limits programmers to write libraries in two ways: it makes some kinds of useful abstractions that inherently require such facilities impossi ble to implement, and it makes other abstractions difficult to implement efficiently. T he OpenC++ MOP addresses this problem by providing li braries the ability to pre-process a program in a context-sensitive and non-local way. That is, libraries can instantiate specialized code depending on how the library is used and, if needed, substitute it for the original user code. The basic protocol structure of the Open C++ MOP is based on that of the CLOS MOP, but the Open C++ MOP runs metaobjects only at compile time. This means that the OpenC++ MOP does not imply runtim e penalties caused by dispatching to metaobjects.

Biographical Sketch Acknowledgments

Shigeru Chiba was born in Tokyo, Japan, in 1968. After liv ing in Osaka This thesis was supervised by Gregor Ki czales. I would like to express my for a few years, he spent most of his childhood in Yokohama, a suburb of deep gratit ud e to him. I also profoundly thank Takashi !l !asuda, my supervi Tokyo. As many other children living in a suburb of Tokyo do, he went to a sor at The University of Tokyo. He continuously supported and encouraged high school in Tokyo, and first met meta computation there when his good me during this work. John Lamping gave me a plenty of constructive com friend at the high school let him borrow a book, Godel, Escher·, Bach: an ments and criticism, whi ch g reatly helped me write this thesis. Finally, I Eternal Golden Braid, by Douglas R. Hofstadter. Looking back now, it was greatly thank my thesis co mmittee, Akinori Yonezawa, 1\ ei Hira ki , Yoshio natural that he was captured by the idea of meta computation, which was Oyanagi, Masami ll agiya, Hiroshi Imai, and aoki Kobayashi. plainly explained in the book. I am grateful for financial support received from J a pan Society for the But the life at university was attractive enough to have him forget meta Promotion of Science (Nihon Gakujyutsu Sinkoukai), from The Information computation. It had actually gone from his mind until it emerged again in technology Promotion Agency, Japan (!PA), from Xerox Corporation, from front of hun at the department of information science. He was supposed to Japan Foundation for the Education and Training of Information Process walk away from it, but what he really did was to dance along a circle and ing (Jyouhousyori Kyouiku Kensyu Jyosei Zaidan), and from International come back to the same place. It might be his karma a lt hough he hopes he Information Science Foundation, Japan (Jyouhou I

6.7 The Standard Template Library. 87 6.8 The OOPACK Benchmark Test 91 6.9 Summary 96

7 Conclusion 99

Contents A Back quote 109 A.1 Quote 109 A.2 Backquote . 109

B Reference Manual 111 Introduction 1 B.1 Introduction . 111 B.2 Tutorial .. 2 Limitations of C++ 112 7 B.2.1 Verbose Objects 113 2.1 Inherita nce 7 B.2.2 Syntax Extension for Verbose Objects 117 2.2 Template 13 B.2.3 Matrix Library 118 2.3 Summary 17 B.2.4 Syntax Extension for the Matrix Library . 121 B.2.5 Before-Method 3 Techniques for Processing a Program 124 19 B.2.6 Wrapper Function 3.1 Lisp Macros ...... 126 19 B.3 Base-Level Language (OpenC++) 3.2 3-Lisp ..... 132 21 B.3.1 Base-level Connection to the MOP 3.3 The CLOS MOP 132 26 B.3.2 Syntax Extensions . . 3.4 Summary 132 35 B.3.3 Loosened Grammar 134 4 The OpenC++ MOP B.4 Metaobject Protocol (MOP) . 13.) 37 8.4.1 Representation of Program Text 4.1 Overview ... . . 13.) 37 8.4.2 4.2 Context-sensitive and No n-local Representation of Types 142 40 4.3 8.4 .3 Representation of Environment 144 Syntax Extension 44 4.4 What is New? . 8.4.4 Class Metaobjects 14.) 46 4.5 Summary B.4.5 Error Message 1.).) 49 B.4.6 C++ Preprocessing 1.)6 5 Meta Helix B.5 Command Reference 1.)7 51 5.1 Implementation Level Conflation 51 c Programs 5.2 Inadeq uate Solutions . . . 159 56 C.l The Distributed Object Library . 5.3 The Meta Helix Architecture 1.)9 58 C.2 The Wrapper Function Metaclass Library 5.4 Implementing the Meta Helix 163 60 C.3 Vector Library 5.5 Summary 166 63 C.4 The Standard Template Library . 168 6 Libraries in OpenC++ C.5 OOPACK benchmark 174 65 6.1 Named Object Libra ry 65 6.2 Distributed Object Library 67 6.3 Wrapper Library 74 6A Implementation of qi\ lake() 79 6.5 Metaclass 81 6.6 Vector Library . . 83

vii

-- ~--·····------~ List of Figures

1.1 The common part and the specialized part of a library 2

3.1 The CLOS MOP is metacircular . 28 3.2 An improved implementation of 3-Lisp 30 3.3 An metacircular implementation 32

5.1 Two handles with the same abstraction 58 5.2 The im plementation relation between interfaces 60 5.3 The Meta Helix supports n implementation levels. 60

6.1 The in stance-of relationship among metaclasses 81

B.1 The OpenC++ Compiler 112 B.2 Instance-of Relationship . 155

-~ ~ --- - - List of Tables Chapter 1

3.1 Lisp Macros, 3-Lisp, and the CLOS MOP . 35 Introduction 6.1 Execution performance of the vector library (J.Lsec.) 86 6.2 Execution performance of STL (msec.) ...... 91 6.3 Execution time of the OOPACK benchmark (sec.) 96 In computing science, elegance is not a luxury 13.1 stati c mem ber functions on Ptree bul a maller of life and dea lh . 136 - E. W. Dijkstra B.2 enum constants returned by Ptree : :Whati s() 138 Execution speed is th e only metric in computer science. - Kei Hirak i

One of today's significant concerns in software industry would be to decrease time and costs of software development. Anybody wou ld agree that good libraries promote code reuse and thereby contribute to rapid and low-cost software development. This thesis deals with a language mechanism for programmers to write good libraries. What are good libraries? Good libraries provide useful control/data al> stractions, which are commonly used for a number of a pplications, hi gh-level to improve readabili ty of programs, and simple and intuitive to avoid lead ing the li brary users to mis use the li bra ries and cause serious errors. Also, efficiency is another criterion of good libraries . Since writing a good library is a very difficult task, the programmer is required to have not only good progra mming skill but also deep knowledge about the application domain in that the libra ry helps progra mm ers. For example, library developers need to know typical functions and data structures used in that domain. Development of good libraries also needs assistance of la nguage design ers, who should provide language mechan isms for writing good libraries. T his is a more realistic option for the designers th a n including all desirable control/data abstractions in the language specifi cations, sin ce the designers cannot ex pect every desirable abstraction in advance a nd because the la n guage would be too complex if built-in language mechanisms cover all the desirable abstrac tions, such as I/ 0, persistency, distribution, concurrency, and so on. Programming languages shou ld have a minimum set of built-in mechanisms a nd most of abstractions should be suppli ed by librari es.

xi 2 CHAPTER 1. INTRODUCTION 3

Motivating problem Solution by this thesis

Language mechanisms that have been developed so far for writing libraries To solve the problem mentioned above, this thesis proposes a new language are not powerful enough to support every desirable control/data abstrac mechanism for enabling libra ries to supply specialized code and include more tions. For example, the C++ language [58] is one of languages that have useful control/data abstractions. With the proposed mechanism, program a richest set of language mechanisms, but a number of useful abstractions mers can build a library consisting of two parts: commonly-used code and cannot be included in a C++ library with satisfying ease-of-use and effi specialized code for every user program (Figure 1.1). The commonly-used ciency. As shown in Chapter 2, C++ programmers cannot write a libra ry code is supplied as is to the user program, but the specialized code is auto class that gives distribution extension to its subclasses. matically generated on demand for a particular user program. The special ized code fill s a gap between the common part and the user program. Our observation on this problem is that such abstractions as distribu The generation of the special iz ed code is programmed by the library tion support are used to give extended features to another abstraction, and developer. The proposed mechanism allows to preprocess a program with thus their implementations are tightly tangled with the implementation of interacting with a collection of code such as class defi nitions and member the other abstraction. Those abstractions need a different implementation function calls. For exampl e, it provides the ability to insert speciali zed code, if they are used wi th a different abstraction. This means that those abstrac to rewrite class defi nitions, to su bstitute different code for member function tions are difficult to include in a library, wh ich is an independent software calls, and so on . component and can provide only a single ge neral-purpose implementation. The language mechani sm proposed by this thesis is the Open C++ MOP There are other kinds of abstractions the implementations of which are [11], which is a metaobject protocol for C++. OpenC++ is the name of a tightly tangled with other parts of the progra m. Abstractions such as a version of C++ language with that metaobject protocol. This mecha nism vector data type can be provided by a library but the implementation that provides class metaobjects, which are regular objects representing a class, so the library can supply is not efficient. An implementation specialized for that library developers can program the generation of specialized code. Al a particular user program can improve the execution performance although though the class metaobjects might seem similar to Smalltalk's class objects that implementation is effective only for the pa rticular program and thus it [26], the class metaobjects receive sou rce code at compile time and prepro- cannot be included in the library. cess it if need ed . The programmers can define a new metaclass (i .e., a cl ass This problem can be avoided if programming languages provide a mech for class metaobjects) a nd thereby they can program desired preprocessing anis m for libra ry developers to supply specialized code through a library to of code involved with the class. The reason that the proposed mechanism is a particular user program. Current language mechanisms , however, allow called a metaobject protocol is that it is essentially a protocol for defining only limited kinds of specialization of li bra ry code. For example, C++'s and accessing metaobjects. Protocol is the Smalltalk terminology a nd it template mechanism allows only type parameterization; library developers means object interface. can specialize only type names appearing in library code for a particular The OpenC++ MOP is a more powerful mechanism than other similar user program . mechanisms like Lisp macros. Unlike Lisp macros, the OpenC++ MOP provides contextual information of the processed code other than a abstract sy ntax tree. The co ntextual information includes the type of a variable, class common part -~ members, a base class, and so forth. This feature makes it possible to per form context-sensitive preprocessing. Furthermore, implementing non-local w }"''"' processing is easy with the Open C++ MOP. Preprocessin g for implementing spec1ahzed part / 8 abstractions often spreads out over the whole source code, but the descrip tion of the preprocessin g invol ved with a single class is centra li zed in to the class metaobject. Since all code fragments are automatically dispatched ac cording to the static types to an appropriate class metaobject, programmers can easily specify preprocessin g that is effective only for a particular class. Figure 1.1 : The common part and t he s pecialized part of a library Preceding techniques known as reflection has a notable inrlu ence on the OpenC++ MOP. Especially, we took the basic structure of the metaobject protocol from the CLOS MOP [36], while we took the basic a rchi tecture CHAPTER 1. INTRODUCTION 5 from Lisp macros. Thus the OpenC++ MOP is also metacircular as the C++ cannot handle. We also mention comparison between the OpenC++ CLOS MOP is, and it is easy to learn and to write an efficient meta-level MOP and other early compile-time MOPs. program. The difference from the CLOS MOP is that the OpenC++ MOP employs static typing and executes class metaobjects only at compile time; Chapter 5: Meta Helix this mea ns that it does not involve runtime penalties due to class metaob jects. Since CLOS is a dynamically-typed languge and thus the CLOS MOP Although metacircularity is a good property, pure metacircular systems can lead to a problem we called implementation level conflation. This problem, executes metaobjects at runtime, avoiding runtime penalties in the CLOS MOP requires complex implementation techniques. found in the CLOS MOP, confuses programmers and often causes program ming errors such as circular definition. To avoid this problem, we - a MOP designer - has adopted an improved version of metacircular architecture for The structure of this thesis the OpenC++ MOP. We present that this improved architecture, named the meta helix, preserves advantages of metacircularity and also addresses From the next chapter, we present background, design detail s, and applica implementation level conftation. tions of the OpenC++ MOP. The stru cture of the rest of this thesis is as follows: Chapter 6: Libraries in OpenC++

Chapter 2: Limitations of C++ The Open C++ MOP makes it possible to include a number of control/data abstractions in a library. We show examples of these abstractions in this We first discuss limitations of current language mechanisms provided by chapter. The abstractions shown here include abstractions that regular C++ C++ for library developers. We illustrate that the inheritance mechanism cannot handle, a metaclass library for helping write a new metaclass, ab and the template mechanism do not work for including some kinds of de stractions implemented by meta-metaclass for facilitating meta-level pro sirable abstractions in a library. And we claim that this problem is caused gramming, and abstractions that the OpenC++ MOP makes more efficient by the lack of the capability of C++ to preprocess a program in a context than in regu lar C++. sensitive and non-local way and instantiate specialized code for a particular program. Chapter 7: Conclusion Finally, we conclude this thesis in Chapter 7. We present contributions of Chapter 3: Techniques for Processing a Program this t hesis and future directions. Next, we overview existing mechanisms that other programming languages provide for processing a program. As the representatives, we show Lisp macros, 3-Lisp, and the CLOS MOP. The feature shared by the three sys tems is that they provide meta representation of programs for the program mers. We overview detailed architectures of the systems and discuss their pros and cons. The bottom line of the discussion is that metacircularity of the CLOS MOP and compile-time executability of Lisp macros are prefer able properties.

Chapter 4: The OpenC++ MOP

On the basis of the discussion in the previous chapter, we proposes a ne\\' C++ mechanism for processing a program. This mechanism is the Open C++ ~lOP and it has the two preferable properties discussed in the previous chap ter: meta.circularity and compile-time executa.bility. The OpenC++ MOP allows programmers to process a program in a context-sensitive and non local way in order to include useful abstractions in a library that regular Chapter 2

Limitations of C++

C++ cannot pre-process a program in a context-sensitive and non-local way. The lack of this capability makes it impossible to include some useful con trol/data abstractions in a library, or makes it difficult to implement some abstractions efficiently as a library. This problem is solved if C++ has a mechanism to process a user program and allow the library to supply special ized code for a particular user program . Single general-purpose codes cannot implement those abstractions or have difficulty in making the abstractions efficient. Existing C++ mechanisms for building a library - inheritance or tem plates - do not provide this capability sufficiently. This means that some practically important abstractions have not been included in a li brary with an ideal interface and efficiency. In fact, some abstractions have been even provided by a specialized C++ language in wh ich the abstractions are em bedded in, or implemented with a dedicated code generator. For example, in the academic world, a number of distributed C++ languages have been developed for making distributed objects available [48, 28]. In industry, pro grammers who want to use distributed objects have needed to write an extra program in an !DL (interface description la nguage) and combine the code generated from the extra program with t heir C++ programs. This chapter presents that C++'s inheritance mechanism or the tem plates mechanism do not work for including some kinds of abstractions in a library. Then it mentions that, to do this, C++ needs a mechanism for processing a user program in a context-sensitive and non-local way. This mechanism makes it possible to develop a li brary including the abstractions t hat regular C++ cannot handle.

2.1 Inheritance

The most basic mechanism C++ provides for library developers is instanti ation. The idea is that the library writer writes a reusable class, which the

I 8 CHAPTER 2. LIMITATIONS OF C++ 2.1. INHERITANCE 9

library user can instantiate. This mechanism is limi ted in that it enables function for marshaling arguments to Move(). This marshaling function con only abstract data types (ADTs); the library user have to use a library class verts the arguments into a byte stream , which lower-level network routines as is. Even if the library class does not exactly fit her requirements, she can directly ha ndle. cannot change the definition of the class at all , and thus she might need to Within the confines of t he inheritance mechanism, however, the class wri te a new class even though a library provide a similar class. This means Distribution cannot supply the marshaling function to the subclass Point. that the reusabili ty of the library is significantly lim ited . Since the marshaling fun ction for Move() performs a kin d of type conversion, Inheri tance is a more powerful mechanism for developing a good li brary. its implementation strongly depends on the signature of Move() , such as It all ows the library user to incrementally define a new class extending a cl ass the number of arguments and their types. But the inheritance mechanism provided by the library. However, the ability of the inheritance mechanism does not all ow the class Distribution to alter the implementation of the to reuse part of a li brary class is significa ntly limited. The inherited code marshaling function to adapt to the signature of Move(). must be always the same; the library cannot s upply different code to different s u bel asses. Named objects This li mitation makes it impossible to develop a library that provides Since the example of the distributed objects is too complex to show the de certain kinds of important abstractions such as distributed objects. These tails here, we in stead use a much simpler example and articul ate li mitations abstractions require that the inherited code be adapted to the user program. of the inh eritance mechanism and what C++ needs to handle some kinds This section discusses this limitation of the inheritance mechanism and be of abstractions that it cannot currently handle. The problem we showed in gins to outlin e what kind of support for adaptation of reused code needs to the example of the distributed objects is that a s uper class cannot supply be provided. some kinds of member functions to a subclass even though supplying them seems desirable from the library users' viewpoint . Distributed objects To discuss this problem, let's implement a simple library with only t he inheritance mechan ism. This simple li brary allows the users to get the class Distributed objects are extended objects, which are accessible over t he net name of an object at runti me. This function is one of req uirements of build work without concern for their location. An ideal libra ry for distributed ing the distributed object library. The users may write something like the objects might allow t he user to write something like t his: followin g program:

class Point : public Distribution { c lass Complex : public NamedObject { public: public: int x, y; double r, i; void Move(int nx, int ny) { x nx; y ny; } }; }; void f(Complex• x) { This program defines a class of distributed objects call ed Point. The class cout << 11 X i s 11 << x->ClassNarne(); Point inherits from the Distribution library cl ass, and this is what gives } makes it be a distributed class. The user can deal with Point objects as regula r objects even if they are on a remote machine: If invoked, f() displays "x is Complex" . The class NamedObject is a li bra ry class, which suppli es a member function ClassName() to Complex. Point• p = Althoug h this pa rticu lar fun ction is already provided in regular C++ p->Move(3, 11 ); by the typeid operator of RTT I [59], we assume that there is no RTTI (Run-Time Type Information) in C++ and t ry to implement this fun ction In this ideal li brary, no special syntax is required to call a member fun ction with the inheritance mechanism. Unfortu nately, this is impossible because for a Point object p that might be on a remote machine. of limi tations of the inh eritance mechanism. for example, the fol lowing Unfor tunately, this id eal class Distribution is not feasible in reg ular definiti on of NamedObj ect works for the class Complex but not others: C++ because it cannot supply all t he member fun ctions that the subclass Point needs to inh eri t . for exampl e, to ma ke t he member function Move() class Nam edObject { call able from a re mo te machine, the class Point needs to inheri t a member public:

- --~- -~ -- 10 CHAPTER 2. LIMI TATIONS OF C++ 2.1. INHERITANCE 11

virtua l char• ClassName() { return "Complex"; } What does C ++ n eed ? } ; The example of NamedDbj ect shows us t hat C++ lacks the ability to al T his version of the class NamedO bj ect s uppli es a member fun ction Class low li braries to s upply code customized for the user programs. With the Name (), which returns a character string "Complex", but this implementa inheritance mechanism, the cl ass NamedDbject cann ot provide an ideal in tion of ClassName () is obviously wrong. If another class Real inherits from terface because it cannot supply a di fferently implemented member fu nction Nam edDbj ect and the member fun ction ClassName() is call ed for a Real ob ClassName() to a subclass such as Complex a nd Real. ject, ClassName () returns an in appropriate cha racter string "Complex". To T his problem is solved if C++ provides a mechan ism fo r programmers to make Class Name() wo rk for the class Real, the implementation of ClassName() program source-code processing and in clude "the program" in a li bra ry. For should be: example, t he developer of t he named object libra ry would wri te a program to process t he user program at source-code level and automaticall y in sert t he virtual char• ClassName () { return "Real"; } implementation of ClassName () customized fo r a subclass such as Complex a nd Real. Then t he library user could avoid implementing ClassName () by However, t he cl ass NamedDbj ect cannot switch the implementations of Cl ass hand for her class. She could write something like t his: Name() to ma ke it work for different subclasses. It has to select either of the class Complex : public NamedDbject { implementations and supply the selected one to all the subclasses. public: We cannot implement the ideal ve rsion of the cl ass NamedDbject in reg double r, i; }; ul ar C++. To make it feasible, we have to ch ange the specifications of the library, but this change also makes the libra ry less easy to use. For the new Note t hat the member fu nc tion ClassName () is no t included by t he defi ni ve rsion of the library, t he definition of the class NamedDbject is as follows: tion of the class Complex since it is automaticall y supplied by the libra ry. T he named cl ass libra ry reads t his program before compilation a nd t rans class NamedDbject { publ i c: lates it in to this program : virtual char• Cl assName() O· II not implemented }; class Comp l ex : public NamedDbject { public: double r, i; The cl ass NamedObj ect does not supply a concrete implementation of Class virtual char• ClassName() { return "Complex"; } Name() any more. ClassName() is implemented by a subclass ofNamedDbject. }; This means that the library users have to implement ClassName() by hand for their classes: The implementati on of ClassName() is inserted by the library. Note that the in se rted implementation is specia li zed for the class Complex. It is not ef class Complex : public NamedDbject { fective for other sibling subclasses like Real. The library class Nam edDbject publ ic: decl ares the member function ClassName() but it does not im plement it: doubler, i; virtual char• ClassName() { return "Complex"; } class NamedDbject { }; public : virtual char• ClassName() 0 · I I no t implemented }; class Real : public NamedObject { public: double value; T his is the same definition that we presented for the feasible but unsatisfyin g virtual char• ClassName() { return "Real"; } ve rsion of the named object library. }; T he macro mechanism might seem an altern ative to t he mechanism pro posed above, but t he mac ro mecha nis m a ll ows ve ry limi ted kin ds of source T hese spec ifi cations of the library are quite un satisfying in term s of ease of code processing, such as simple wo rd -by- \\'ord repl acement and concatena use. Note t hat the defini tions of t he user classes Complex and Real now tion of words. To implement t he named objec t li bra ry wit h ideal interface, incl ude t he im plementation of the member fun ction ClassName(). In the C++ is required to have a more powerful and soph isticated mechanism, ideal s pecifi cations, this should be supplied by the libra ry cl ass NamedDbj ect. which should satisfy the followin g t wo cri teri a: 12 CHAPTER 2. LIMITATIONS OF C++ 2.2. TEMPLATE 13

• Context-sensitive class Complex : public NamedObject { public: How a program is processed should be determined with referring various doubler, i; contextual information of the processed program. The contextual informa EXPAND_CLASSNAME_HERE(Complex) tion includes what RTTI (Run-Time Type Information) already provides }; but not limited. It is a class definition, type information, program text, and The macro function is EXPAND_CLASSNAME_HERE, which is expanded into an so on. Providing contextual information means that the information can appropriate implementation of ClassName(). The library users have to ex be used even if it is defined at other locations than where the source-code plicitly insert the macro in the definition of all the subclasses ofNamedObj ect, processing happens. For example, if an assignment expression to a variable but inserting it in all the places by hand is quite error-prone. is processed, the type of the variable may be declared at a different location. Context-sensitivity is a crucial property of the proposed mechanism since the processing by the macro mechanism is independent of the contextual in 2.2 Template formation of the processed program. Context-sensitivity gives an advantage against the macro mechanism to the proposed mechanism. In the previous section, we presented that a library including some kinds of In the example of the named object library, the library has to examine abstractions need to supply customized code for a particular user program. the name of the subclass that inherits from the library class NamedObject. The template mechanism of C++ achieves this ability to a certain degree, This name is directly embedded in the implementation of the member func but it cannot be a general solution of our problem. In this section, we tion ClassName(), which are inserted in the definition of the subclass. The mention the ability that the template mechanism provides and then we use of contextual information is more significant in the example of the dis present why the mechanism is not a general solution. tributed objects. For example, to supply a marshaling function, the library has to examine the signature of a member function and specialize the imple Vector mentation of the marshaling function for the particular member function. With the template mechanism, programmers can write a library that supply customized code for a user program although the range of customization is • Non-local limited. For example, the template mechanism enables a library for a vector The proposed mechanism should make it easy to program not only local abstraction. With this library, the users can write a program like this: processing but also non-local one, which needs to deal with many code frag Vector vi, v2, v3, v4; ments spread out in various locations of the processed program. For ease of programming, locations where the processing must happen should be spec v1 = v2 + v3 + v4; ified in a declarative way. If the programmer has to explicitly specify every location, the programming would be extremely difficult and unrealistic. The variables v1, v2, v3, and v4 are vectors of integers. Using vectors for For the example of the named object library, the library has to read other types is also easy. If the users want to deal with vectors of characters, the whole program and insert the member function Cl assName() in the they should say Vector instead of Vector. definitions of all the subclasses of NamedDbject. The mechanism should The inheritance mechanism does not enable such a useful vector library. help the programmer easily specify locations to insert ClassName (). For If the vector library is implemented with the inheritance mechanism, the example, it should allow the programmer to direct something like "insert library users need to define a new subclass for every vector for a different ClassName () in a class declaration if the class is a subclass of NamedOb j ect." type. This results from the same reason why the named object library is C++'s macro functions are rather a mechanism for local processing. It not feasible; to make vectors for various types available without subclassing, processes only places \vhcre a macro name appears, and so programmers the vector library has to alter the implementation, depending on whether have to place the macro name by hand wherever the processing is needed. the vector is for integers or characters. For example, if the named object library is developed with a macro function, The template mechanism can absorb the difference of implementations if the library users would need to write an awkward program like t he following: the difference is type names. It directs the C++ compiler to automatically produce the implementation adapted for a particular user program. The following program is the definition of the vector abstraction, which should be included in the vector library: 14 CHAPTER 2. LIMTTATJONS OF C++ 2.2. TEMPLATE 15

template class Vector { To compute v2 + v3 + v4, this implementation eventually needs two T elements[SIZE]; public: for loops from the first through the last element of the vector, but executing Vector operator + (Vector& a, Vector& b) { this loop twice is redundant. The whole expression should be computed by Vector c; the following more efficient implementation: for(i = 0; i < SIZE; ++i) c.elements[i] = a.elements[i] + b.elements[i]; for(i = 0; i < SIZE; ++i) v1.elements[i] = v2.elements[i] + v3.elements[i] return c; + v4.element[i]; } This implementation executes the loop only once, and it directly sets the }; summation of the three elements to v1. This inefficiency is not due to the mechanism of operator overloading. For simplicity, this example assumes that the length of a vector is always Rather, it should be thought that is caused by limitations of the template SIZE. In the Vector template above, Tis a type parameter. All occurrences mechanism. Operator overloading is a mechanism for syntax sugar, and ofT in the template are replaced with an actual type given by a user pro the problem is not solved even though various kinds of operators can be gram when the compiler produces the actual implementation of the vector overloaded. For example, suppose that the programmer can overload the abstraction. For example, the implementation that the compiler produces three-operands + operator. Then the developer of the vector library would for Vector is equivalent to this (pseudo) class definition: include the following code in the library:

class Vector { template int elements[SIZE]; Vector operator + (Vector& a, Vector& b, Vector& c) public: { Vector operator + (Vector& a, Vector& b) { Vector v; Vector c; for(int i = 0; i < SIZE; ++i) for(i = 0; i < SIZE; ++i) v.element[i] a.element[i] + b.element[i] c.elements[i] = a.elements[i] + b.elements[i]; + c. element [i] ;

return c; return v; } }

}; This operator function executes such an expression as v2 + v3 + v4 more efficiently, but the library is still inefficient to deal with other kinds of expres Limitations of templates sions such as v2 + v3 - v4 and v2 + v3 + v4 + v5. The library developer cannot overload all combinations of operators in advance. The vector abstraction can be included by a library with ideal interface if the template mechanism is used, but this implementation of the vector ab C++ needs a mechanism for processing a program straction is not satisfying with respect to execution performance. Consider The template mechanism cannot enable the efficient implementation, which how this expression is executed if the vector abstraction is implemented as we showed above: executes a for loop only once. This is because of the limitations of the ability of the template mechanism to supply code adapted for a user program. The v1 = v2 + v3 + v4; only adaptation that the template mechanism can perform is to simply fill out parameterized fields in the template with given type names.' This is Since the + operator is overridden for Vector, the execu tion of this not sufficient to implement the vector abstraction efficiently. expression is divided in to two function calls, each of which executes a for In general, the template mechanism is not suitable for this kind of inter procedure optimization which needs to process seve ral operations at a time loop to compute an addition of two vectors. The called fun ction receives 1 two vectors, executes t he for loop to compu te the addition of each vector a nd substitutes specialized code for all the operations. To do this, C++ element, and returns the result. 1 A template parameter may be not. only type names but also any constant value. 16 CHAPTER 2. LIMITATIONS OF C++ 2.3. SUMMARY 17

needs to be able to handle context of the processed operations: what the complex flow-analysis. An advantage of our proposed mechanism is that it group of operations are computing. If C++ has a mechanism to process a allows libra ry developers to implement ad-hoc optimization which is obvi program with that context sensitivity, then the user program: ous to the developers but difficult for compilers to automaticall y perform. The libra ry developers can include the program for source-code processing Vector vl, v2, v3, v4; in a library so that a user program using the library is pre-processed and efficiently compiled by a back-end compiler. v1 ; v2 + v3 + v4;

can be translated by the vector library before compilation into: 2.3 Summary

Vector vl, v2, v3, v4; This chapter mentioned that some useful abstractions cannot be included in a library, and others are difficult to efficiently implement within the confines for(i ; 0; i < SIZE; ++i) of C++. Such abstractions require different implementations for different vl.elernents[i] ; v2.elernents[i) + v3.elernents(i] library user programs, but existing C++ mechanisms do not provide the + v4.el ernent[i]; ability to do that enough to implement those abstractions. After the translation, the efficient loop is substituted for the vector expres In this chapter, first, we showed that the inheritance mechanism does not sion . The overloaded + operator function is not called any more, but the enable developers to implement the distributed object library or the named additions are computed by the inlined loop. object li brary. As for the distributed object library, the developer cannot define a library class Distribution so that it supplies subclasses with a Context-sensitivity and non-locality are also significant for this process ing as in the example of the named object library. First, context-sensitivity mars haling function, which needs to be differently implemented for different subclasses. This is because the inheritance mechanism forces a library class is needed to determine which expressions should be translated. Since the translation is applied only to vector expressions, the library needs to look up to supply member functions to subclasses as is without any adaptation. the type of a variable in an expression and determine whether the translation Then, we presented that the template mechanism provides the limited is applied or not. Second, this processing is non local; All vector expressions ability to supply different implementations for different user programs, but in the whole program needs to be found and translated into efficient code. this ability is not powerful enough to implement some abstractions such as Without appropriate supports for programmers to specify places at which the vector abstraction. Because of the ability to supply different implemen the translation should be done, programing the translation for the vector tations, the vector library implemented with the template mechanism allows library \\'Ould be difficult and awkward. the users to easily deal with vectors for various types. However , this vector library is not efficient because of the limitations of the ability of the template mechanism. If the library can su pply cleverer implementations customized Can optimizing compilers do the same thing? for a particular use r program, the provided vector abstractions would be In the example above, we merged two distinct function calls and inlined more efficient. the resulting optimized loop. This way of performance improvement is re To solve the problem above, this chapter claimed that C++ needs a garded as optimization that regular C++ compilers can perform. As \\'e more powerful mechan is m for processing a program at source-code level. If show in Chapter 6, however, the inter-procedure optimization seems difficult this mechanism is amilable, programmers can develop a library that pre for practical compilers to perform within reasonable time and space. Since processes a user program and inserts code needed for implementing an ab inter-procedure optimization requires deep flow-analysis and very clever code straction in an adapted way to the user program. Through the discussion generation, \\'e s hould not expect that compilers perform all possible inter with the concrete examples, we presented that the proposed mechanism procedure optimization in general. What we can expect is that compilers should have two important properties: context-sensitivity and non-locality. may perform the optimization for some typical patterns of program. First, a user program should be processed in a context-sensitive way. For Although inter-procedure optimization is difficu lt for C++ compi lers, example, to produce a marshaling function, the distributed object library it is often obviou s and straightforward to perform from the programmer's needs contextual information of a use r program such as sign(l.tures of member viewpoint. Since the programmers know semantic information of the pro functions and type information of variables. Second, the proposed mecha gram, they can easily find possible inter-proced ure optimization without ni sm shou ld make it easy for developers to program non-local processing, 18 CHAPTER 2. LIMITATIONS OF C++

which deals with code fragments spread out over the user program. For ex ample, the efficient vector library has to find all vector expressions included in a user program and translate them into efficient loops. Without appro priate supports, programming such non-local processing wou ld be difficult and error-prone. Chapter 3

Techniques for Processing a Program

The previous chapter presented that C++ needs a more sophisticated mech anism for processing a program . T he existing mechanisms, the inheritance mechanism or the template mechanism, do not enable context-sensitive or non-local processing. This chapter overviews currently known mechan isms for program processing and mentions their pros and cons. The basic idea shared by these mechanisms is to provide the meta representation of the programs. The meta representation gives programmers the capability for context-sensitive and non-local processing, which we need for C++. As the representatives of these mechanisms, this chapter shows Lisp macros, 3-Lisp, and the CLOS MOP. We compare the meta representation they provide, and discuss pros and cons. The focus of t hi s chapter is on illustrating essential ideas behi nd t he mechanisms rather than showing the exact specifications. Hence the description in this chapter is not exactly faithful to t he original syntax or specifications. We carefu ll y alter the syn tax and t he specifications so as to help clarify the differences a mong the mecha ni sms but not to lose the essence of their ideas.

3.1 Lisp Macros

Un li ke C++ macros, wh ich perform simple word-based replacement ignoring the syntax, Lisp macros allow the programmers to manipulate program text as data; A program is manipulated through an ordinary data structure. We ca ll this data structure a meta representatio11 of the program. In Lisp mac ros, the meta representation of a prog ram is an abstract syntax Lree. T he program text is represented in the form of t he tree whose leaf nodes are the lexical tokens of the program text. Lisp macros enable programmers to implement some kinds of abstrac tions that Li sp functions ca nnot implement. Those abstractions are called

~ ------~- 20 CHAPTER 3. TECHNIQUES FOR PROCESSING A PROGRAM 3.2. 3-L/SP 21

special forms in the Lisp terminology. For example, the following macro replacement of type names. On the other hand, Lisp macros cannot han implements a special form rbegin that sequentiall y evaluates expressions dle context-sensitivity or non-locality and thus just porting the Lisp macro from right to left (in the reverse order of begin) and returns the resulting system to C++ does not solve our problem. valu e of the leftmost expression :

(define rbegin 3.2 3-Lisp (macro exprs '(begin .~(reverse exprs)))) Lisp macros provide the meta representation of programs so that they can implement some kinds of abstractions that Lisp functions cannot implement. A backquote (') and .~ are convenient notation for constructing a tree Although the meta representation in Lisp macros is on ly program text, 3- structure. If the readers are not familiar to this notation, see Appendix A). Lisp [54, 53] provides not only program text but also other information This macro is used as follows: as the meta representation. This feature of 3-Lisp enhances the variety of abstractions that programmers can implement. (rbegin (+ 3 4) (list 1 2) C• 5 8))

This program is processed by the macro function rbegin before being ex Meta representation ecuted. The macro function is an ordinary Lisp function except that it The meta representation of programs in 3-Lisp consists of program text, receives and returns program text. The macro argument exprs is bound to the current environment, and the current continuation. Adding the latter a list ((+ 3 4) (list 1 2) C• 5 8)). Then the macro function returns two makes it possible to implement abstractions that Lisp macros can not this program: handle. Suppose that we implement a special form defined?, which returns true if the given variable is defined in the current environment: (begin (• 5 8) (list 1 2) (+ 3 4)) > (define x 1) This is substituted for the original program (rbegin ( + .. . ) before the X execution. The resulting value of the program is that of this substituted > (defined? x) #t program, that is, 7. > (defined? y) Note that a Lisp function reverse is call ed during the macro expansion. #f It reverses the order of the li st th at exprs is bound to. Since the processed program text is the first-class data represented in the tree structure, 1 this The special form defined? returns #t (true) for the variable x, but #f (false) Lisp function can process it as it processes ordinary Lisp data. This capabil for the variable y. ity makes Lisp macros different from other simple macros like C++ macros. Because Lisp macros cannot examine the current environment, they can Indeed, C++ macros cannot implement the special form rbegin since the not deal with this special form. However, 3-Lisp can do. See the following given program text is not the first-class data. They can only perform the program written in 3-Lisp:2 limited operations on the program text. Only word-by-word replacement (define defined? and concatenation are allowed. (meta (expr env cont) (if (is-bound? env (car expr)) Applicability to our problem (cont #t) (cont #f)))) Lisp macros give a partial solution of the problem we discussed in the pre vious chapter. They allow programmers to process a piece of code following The special form defined? is implemented by a meta function (originall y a macro name. An advantage of Lisp macros is that programmers can pro called lambda reflect in 3-Lisp). Unlike macro functions, which receive only gram how the piece of code is expanded. This provides the ability to generate program tcxt 1 the meta function receives the current environment env and more specia li zed code than what the template mechanism can do by simple the current continu atin cent as well as the program text. expr. The cu r rent environment represents the bindings between symbol names and values, 1 The first-class data are t.hc data that the program can handle as th e object or the 2 computation. f'or· example, numbers, symbols, lists, and vectors, are the first-class data To make it easy to read, we modify the syntax of 3-Lisp. We believe that this modi in Lisp. ficat ion docs not afrect the essential idea of 3- Li sp but rather helps articulate it,. 22 CHAPTER 3. TECHNIQUES FOR PROCESSING A PROGRAM 3.2. 3-LJSP 23

while the current continuation represents the control flow after this special (if (null? rest) form finishes. Note that the meta function may access and change the en (c , ???) (loop (cdr rest) vironment and the continuation sin ce they are the first-class data within (lambda (r) the meta function. In fact, to implement defined?, the meta function calls (eval-expr env environment a built-in function is-bound? and looks up a symbol name obtained by (car rest) sub-expression (car expr) in the received environment env. Then it calls the received c))) continuation continuation cant with #t or #f, which is the resulting value of the special form. 3 Note that a built-in meta function eval-expr (normalize in the 3-Lisp ter minology) is called to interpret each sub-expression of rbegin. According to the given sub-expression, eval-expr causes side effects on the environment Base level and meta level and calls the continuation with the resulting value of the sub-expression . Meta functions are not extended macro functions that receive more argu ments. There are significant difference between macro functions and meta Self modification functions. Meta functions run at runtime because they manipulate the cur [n the examples shown so far, we have in troduced new keywords such as rent environment and the current continu ation, which are only available at rbegin and defined?, and defined their meanings to implement new ab runtime. Hence a meta function directly interpret the received program text. stractions. Some kinds of abstractions, however, requ ire altering the mean The value returned by the meta function is the result of the interpretation. ings of the existing keywords or syntax. In other words, they need to modify The meta function also may cause sid e effects on the environment as the the default behavior of the language rather than to extend it. A feature of result of the interpretation. On the other hand, macro functions may run this self modification is that the effects of the modification are applied to pro at either runtime or compile time since they deal with only static informa grams even though the programs are not edited. Editing them or inserting tion of the program, that is, program text. They are functions that receive new keywords are not necessary. program text and transform it, but they do not directly interpret it. The Some followers of 3-Lisp, such as Black [2], enables the self modification. value returned by a macro function is the transformed text. They provide many built-in meta functions, which carrry out a primitive In 3-Lisp, both ordinary functions and meta functions run together at base-level operation, and a ll ow programmers to red efine them to change the runtime. To distinguish the two kinds of functions, 3-Lisp has two execution default behavior of the operations. For example, they may alter the behavior levels, which are the base le vel and the meta level. The two execution levels of the language when reading a variable. By default, reading an undefined are id entical except that the objects of the computation at the meta level is variable causes an error. We alter this behavior so that 'undef is returned the interpretation of the base-level program. The result of the computation if the variable is undefined: at the meta level reflects on the computation at the base level. This relation between the meta level and the base level is called causal connectivity. Fo r (define change- eval-read (meta (expr env cant) example, meta functions may change the environment at the base level in (set ! eval-read order to define a new symbol name, or to change the value that a symbol (lambda (expr env cant) name is bound to. (let ((var (car expr))) !\ [eta functions can use a built-in meta function if they interpret the (if (is-bound? env var) (cant (lookup env var)) received program text in the default way. For example, the program below (cant 'undef))))) is another implementation of the special form rbegin with a meta function: (cant #t)))

(define rbegin If the meta function change-eval-read is executed, it calls set' to (meta (expr env cant) (let loop ((rest expr) substitute a new meta function for the built-in meta function eval-read. (c cant)) Note that set ! has to be invoked at the meta level. Otherwise, set! wou ld replace a base- level function eval-read, if any, with the new one. The 3 1n reality, the base~le\'el value has different representation at the meta level. For substituted meta function first checks whether the given variable is defined example, #t (a boolean) becomes '#t (a symbol) at the meta level. Therefore, the meta or not in the current environment. If not, it uses 'undef for the value of function mu st con\'ert # t into the meta representation before passing it to the continuation cont. t he variable: 24 CHAPTER 3. TECHNIQUES FOR PROCESSING A PROGRAM 3.2. 3-L/SP 25

> X 3-KRS is designed mainly for customizing the default behavior of the ERROR: undefined symbol language on demands, rather than implementing new abstractions on top of > (change-eval-read) #t the language. In 3-KRS, each base-level entity such as objects a nd messages > X is associated with a special object called metaobject. The metaobject is UNDEF the meta representation of the base-level entity. Calling a method for the metaobject, programmers can obtain the meta information of the entity. This sort of self modification is difficult for Lisp macros. Some imple For example, they can inspect what methods an object has through the mentations of the Lisp macro system all ow a macro function to override an metaobject for the object. Also, the metaobject provide similar capability existing keyword. For example, programmers may define a macro fun ction of a built-in meta function in 3-Lisp. For example, t he metaobject for an named if to alter the behavior of the if special form. However, Lisp macros object has a method for invoking a method on the object. cannot handle such modification t hat we showed in t he change- eval-read As in 3-Lisp, programmers can alter the default behavior of 3-KRS. example because the expression for reading a variable is not preceded by In stead of redefining built-in meta fun ctions, the programmers define a new any keyword. To process an expression by a Lisp macro, programmers have metaobject to implement the new behavior. Suppose that they alter the to expli citly place the macro name in front of the processed expression. default behavior of t he object creation so t hat the history of object creation To use a Lisp macro an d get t he same result that change-eval-read is reco rd ed. Wi th this language customization, programmers may write provides, programmers have to use a programming convention when read something like this: in g a vari able. For example, they may have to write (read- variable x) > *history* instead of just writing x, whenever reading a variable x. This program () min g co nvention makes it possible for a macro read-variable to process > (defclass point () (variable x y) :meta recorded-object) t he expression x. Another approach is to su rround the whole program by a POINT macro call. This technique is called t he code walker. The macro function > (defclass rect () receives the whole progra m as t he argu ment, looks up ex pressions for read (variable top bottom left right) :meta recorded-object) ing variables, and translates the expression s to alter the behav ior. Writing RECT > (define p (make-instance point)) the code walker is not difficu lt in Lisp because the grammar is simple, but p it is difficult in other langu ages such as C++. > (define q (make-instance rect :meta recorded-object)) Q > •history* Reflective languages (RECT POINT)

3-Lisp is one of the earliest langu ages t hat can handle richer meta represen If a class annotated with ":meta recorded-object" is instantiated, the tation of the programs than Lisp macros.'' 3-Lisp has bee n followed by many class name of the instance is added to t he li st indicated by •history•. For la nguages, and this famil y of languages a re often called reflective languages the example a bove, since a point object and a rect object a re created, the or languages with a meta archi tecture. For example, Brown [60], Blond class point and the class rect are added to t he list. [18], and Black [2] a re Lisp-based successors of 3- Lisp. These successors To implement this language customi zation, the programmer first defines has been developed to stud y the semantics a nd the implementation of the a new cl ass recorded-object for metaobjects:5 infinite tower of the execution levels, that is, the base level, the meta level, (defclass recorded-object (metaobject-for-object) the meta-meta level, a nd so on. This was also one of t he main iss ues of t he (defmethod create-object (expr env) study of 3-Lisp. (let ((class-name (car expr)) T he meta archi tecture developed by 3-Lisp has been also a pplied to (h (env-ref env '•history•))) 1 object-oriented languages. Carl y representatives of s uch languages are CLOS (env-set env '•history• (cons class-name h)) (< - super create-object expr env)))) [56, 36] a nd 3- h:RS [38]. The two languages have bee n developed und er di fferent design strategies. Since we discuss C LOS in t he next section , we The class r ecorded-object inherits from the default class metaobject introduce o nl y 3-KRS here. for-object and ove rrides a method create-object to maintain *history*. 5 4 ln Artifi cial lntelti gcncc, a similar idea was proposed earli er than 3-Lisp.[3, 25, 62) Again, we use an al tered version of 3- I

The method create-object is invoked when an object is created. The new base-level and meta-level programs are written in the same syntax, but the create-object first adds the cl ass name to the list indicated by the base two "languages" for the base-level and meta-level programs are not identical. level variable +history•. Then it calls the overridden method of the super This section discusses {truly) metacircular reflective languages repre class metaobj ect-for-obj ect, which creates a new object in the default sented by the CLOS MOP. Unlike 3-Lisp or 3-KRS, its base level and meta way. Note that create-object has to call bu il t-in meta functions env-ref level programs are written in an identical language. This means that the and env-set 1 to access the base-level variable +history•. It cannot di customization by the meta-level program affects not only the base-level lan rectly access this variable since create-object is at the meta level. Any guage but also the meta-level language in which the meta-level program is base-level entity must be dealt with through built-in meta functions. written. This feature gives some benefits to reflective languages. Other object-oriented reflective languages include ABCL/R [61, 41, 40], Ferber's language [21], RbCI [32], The MIP for C++ [9], OpenC++ ver Metacircularity sion 1 [13], AL-1/D [46, 45], CodA [42], a reflective version of BETA [6], and Iguana [29, 30]. They have explored various applications of the meta Metaobject protocols {MOPs) are another name to indicate a meta architec architecture. ABCL/R is a parallel language, and it allows programmers ture such as 3-Lisp's one. Particularly, MOPs mean programming interfaces to customize the default schedulin g policy. OpenC++ version 1 enables for customizing the language. The word "metaobject protocol" has been programmers to implement language extensions such as distribution, per first used in CLOS (Common Lisp Object System) [56] . The CLOS MOP sistence [57], and fault-tolerance [20], within the confines of the language. [36] enables programmers to incrementally customize CLOS in CLOS itself AL-l/D's application is similar to ABCL/R. It allows programmers to cus with a meta architecture. tomize the policy of object migration. Coda employs the meta architecture A unique feature of the CLOS MOP is that the system is metacircular to run the same Smalltalk program on different platforms. For example, if (Figure 3.1) . In 3-Lisp, meta functions and base-level functions are written the programmer ports a program written for a si ngle processor machine to in the same language, but the two "languages" for the meta and base-level run on a multi-processor machine, she has only to define new metaobjects. functions may not be identical; they may be two distinct instances of the Sin ce the platform-dependent code is separated into the metaobjects, the same language. At beginning, the two languages are identical and are thus programmer does not need to edit the base-level program. meta-circular in some sense. But once the base language is customized, the meta language is left unchanged and the two languages are therefore different. Applicability to our problem This fact would be clear if we reason about the object of language cus Reflective languages represented by 3-Lisp show us how meta representation tomization; the language customized by a meta-level program is just the of a program should be exposed to the programmer. Especially, object language for a base-level program. For example, a new special form defined ori ented reflective languages s uch as 3-KH.S shows that metaobjects can be by a meta function is only available for base-level functions but not available good abstraction to deal with complexity of the meta representation. for other meta functions. The customization by meta functions reflects only However, reflection cannot be a solution of our problem since program on t he language for base- level functions, but it does not circularly reflect on mers describe interpretation of a base-level program to define a new mecha the language for the meta functions. nism or change an existing mechanism. Although programmers can describe The reason for this non-metacircularity would be to avoid apparent in interpretation that causes the same effects that we want, describing source finite regression caused by a circular definition. If 3-Lisp is naively made code processing is more intuitive and straightforward as for our motivating metacircular, programmers would easily define a meta function with using appli cations. Also, runtime penalties implied by the in terpretation is an the special form defined by the meta function itself and the s pecial form other problem of reflection. would cause infinite regression. But th is problem can be avoided with keep ing metacircularity if another scope control mechanism is introduced. To keep metacircularity while letting programmers avoid a circular def 3.3 The CLOS MOP inition , the CLOS MOP uses a class system for controlling the scope of language customization. l-Ienee both base- level programs and meta-level We claim that the reflective languages introduced in the previous section, programs are written in the same in sta nce of the language) and language such as 3-Lisp and 3- I

the CLOS MOP, this customization is implemented by this program:7

Meta-meta level Meta level (defclass recorded-class (metaobject-for-class) Meta level (defmethod create-object (init-args) t 0 Base level (let ((class-name (<- self name))) Base level (set! •history• (cons class-name •history•)) (<- super create-object init-args)))) 3-Lisp TheCLOS MOP Since the CLOS MOP does not provide the metaobject for an object, the pro Figure 3.1: The CLOS MOP is metacircular gram defines a new metaclass, which means a new class for class metaobjects. It inherits from the default class metaobject-for-class and overrides a method create-object. When an object is created, the new create-object first calls a method name for the metaobject (self) to obtain the class semantics of the customized language and the meta-level program customiz name. Then it updates the history of object creation and calls the over ing the language. ridden method for the super class, which creates an object in the default Despite of the metacircularity, CLOS programmers can avoid a wrong means. 6 circul ar definition. In the CLOS MOP, customization is specifi ed on a class Because of metacircula rity, there is no explicit (syntactical) distinction basis; it is applied only to particular classes and their instances. Thus, if between base-level programs and meta-level programs in the CLOS MOP. programmers carefully distinguish classes, they can avoid a circular defi The two kinds of programs are written in the same language and run under nition. Suppose that a meta-level program alters the behavior of a class the same runtime environment. They may even coexist in a single fun ction Point. Since the CLOS MOP is metacircular, the programmer may use the or method. Therefore, metaobjects in the CLOS MOP can directly access customized class Point in the meta-level program, but she can still avoid a the base-level data. The metaobjects can use the base-level primitives to circu la r definition unless she explicitly implements the customization with access the base-level data as the objects can do. There is no difference in Point to be a circular definition. primitives that the metaobjects and the objects can use. For example, the The origin of the metacircular architecture would be found in Smalltalk- method create-object directly reads and writes the variable •history• 80 [26]. In fact, Foote reported the use of the metacircularity in Small talk- · with base-level primitives like set!. Recall that create-object in 3-KRS 80 in his paper [22]. Also, CommonLoops [4] shou ld be noted as an early has to access through built-in meta functions env-ref and env-set! since metacircular language. Furthermore, the metacircular architecture is found the base-level objects and the metaobjects are run at distinct execution in ObjVIisp [17], Classtalk [8], EuLisp [7] and STklos [24] . levels.

Direct access to the base level Ease of learning

The CLOS MOP provides the capability for customizing the behavior and Base and meta levels in non-metacircular systems are written in different implementation of the object system of Common Lisp (i.e., CLOS) . For languages. This separation makes access across levels be complex and tend example, programmers may define a new metaobject for classes to alter the to be inefficient. On the other hand, the metacircularity of the CLOS iV!OP rule of multiple inheritance. This customization makes it easy that CLOS avoids gratuitous differences between levels, while sti ll being effective, and runs programs written in other Lisp-based la nguages such as Loops [5] and gives a few advantages. Flavors [10]. First, the meta architecture employing metacircularity is easy to learn. Despite the metacircularity, the way of customization with the CLOS Since programmers can use base-level primitives to access base-level data i\ IOP is quite similar to the way of customization in non-metacircular lan from the meta level , they do not have to learn built-in meta functions unless guages s uch as 3-1\RS. In the previous section, we s howed the 3-I

7 6 T he solution by the CLOS MOP is not.. a com pl ete solution. We revisit this issue in Again, we use an altered sy ntax for emphasizing differe nce from the 3- J

switch interpreter the default interpreter are not identical; at least, they arc distinct instances of an interpreter and they maintain different runtime environments. Because default interpret I compile interpreter t the default interpreter does not handle meta functions, it can achieve as good meta functions efficiency as an optimized interpreter for ordinary Lisp. Or, we may use a compiler for better efficiency instead of an interpreter . .-~--=----.------! interpret To use the default interpreter from a meta function , programmers need Base-level program to call a built-in meta function like eval-expr and ex plicitly switch to it. Recall the implementation of create-object in 3-KRS. It can be rewritten to explicitly switch to the default interpreter: Figure 3.2: An improved implementation of 3-Lisp (defmethod create-object (expr env) (let• ((class-name (car expr)) (expr2 '(set! •history• For example, a metaobject in the CLOS MOP can call a method for (cons ,class-name •history•)))) an object in the same syntax that an object calls it at the base level. The (eval-expr env expr2) metaobject uses a built-in meta function only when it needs to access a meta (<- super create-object expr env))) representation of the program, for example, the class name of an object. Recall that create-object of recorded-class executes (<- self name) The built-in meta fun ction eval-expr executes the expression expr2 with to obtain the class name. On the other hand, a metaobject in 3-KRS has to the environment env by the default interpreter, so that the expression is call a built-in meta function even when it calls a method for an object. In executed faster. For example, since the expression accesses the base-level 3-KRS, any object must be indirectly manipulated through the metaobject variable •history• twice, the default interpreter may memoize the memory associated with that object. To call a method for the object, an appropriate location of the variable and reuse it for the second access. However, the built-in meta function for the metaobject has to be called. optimization that the default interpreter can do for eval-expr is limited to runtime optimization since eval-expr receives an expression and an envi Ease of implementing efficiently ronment at runtime. Even if an compiler is used instead of the interpreter, no static optimization is applicable. Another advantage of metacircularity is execution performance. Metacir On the other hand, metacircular systems naturally give more freedom cu lar systems make it intuitive and straightforward for implementors to to the default interpreter or compiler. For example, in the CLOS MOP, develop an efficient interpreter and compiler. To do the same thing in non there is no explicit distinction between the base le vel and the meta level, so metacircular systems like 3-Lisp, they need sophisticated implementation the default interpreter can execute both base-level a nd meta-level programs techniques or they need to force programmers to use advanced program (Figure 3.3). This means that programs implicitly switch direct interpreta ming techniques. tion by the default interpreter and double interpretation via metaobjects. If naively implemented, 3-Lisp is extremely slow. The typical imple Recall the implementation of create-object in the CLOS MOP: mentation of 3-Lisp uses two interpreters for keeping two distinct execution (defmethod create-object (init-args) levels. The first interpreter executes the second interpreter, while the second (let ((class-name (<- self name))) interpreter executes base-level functions. In this implementation, meta func (set! •history• (cons class-name •history•)) tions are regarded as part of the program of the second interpreter. They (<- super create-object init-args))) are executed by the first interpreter and execute base-level functions. This double interpretation maintains the causal connectivity between the base This method directly accesses a base-level variable •history• with the base level and the meta level, but it sign ifi cantly decreases the execution speed level primitive set!. Since the expression (set! ... ) is written in the of the base-level program even though no meta functions a re used. sy ntax for the base-level program , it can be directly executed by the defau lt Improvin g the execution performance is relatively easy if the base-level interpreter, and hence the system implicitly switches to direct execution by program does not use meta functions. We can prepare the third interpreter, the default interpreter. named the default interpreter, and switch to it while the base-level progra m This implicit switching gives a lot of room for opti mi zation especiall y runs without meta functions (Figure 3.2). Note that the first interpreter and to a compiler. Unlike the way by usin g eval-expr, an executed expression 32 CHAPTER 3. TECHNIQUES FOR PROCESSING A PROGRAM 3.3. THE CLOS MOP 33

The underlined functions are built-in meta functions. The above imple default interpreter mentation explicitly compiles an expression and memoizes it when it is first executed, and reuses the compiled code from the second time. Although this switch ...._ I illterpret I compile ,_,. implementation will run a bit faster, the built-in meta functions for opti interpret ' /compile metaobjects mization make the language complicated and difficult to use. Programmers j have to learn how the built-in meta functions help for optimization. ------*illferpret r==aase-level objects ~ When metaobjects run Normally, metaobjects in metacircular systems run at runtime as in non Figure 3.3: An metacircular implementation metacircular systems. Metacircularity has nothing to do with when metaob jects run at runtime or compile time. However, because running metaobject at runtime impairs execution performance, the actual CLOS MOP employs is statically given to the default interpreter and hence various optimization a technique called currying8 so that the metaobjects less frequently run at techniques using static information are naturally applicable. runtime. This technique is based on the observation that some of compu To get equ ivalent effects in non-metacircular systems, a sophisticated tation by metaobjects can be statically performed, or, once it is done, the optimizing compiler or complex programming techniques are needed. For result can be memoized and it does not need to be performed again. So example, an optimizing compiler using a technique called partial evaluation the currying technique explicitly splits the protocol into one for com puta [19, 23) or inlining will compile the following program in 3-I

method. lookup first checks methods directly supplied by the class, and, if not found, it searches su per classes. Table 3.1: Lisp Uacros, 3-Lisp, and the CLOS MOP This implementation is not effici ent because the targeted method is looked up in the class hi erarchy for every call. It should be looked up only Lisp Macros 3-Lisp CLOS MOP once when it is first call ed, and the found method s hould be memoized and Meta representation 6 0 0 reused for all su bsequent calls to the method. Self modifi catio n X To avoid this inefficiency, the actual protocol has been design ed with the 0 0 cu rrying technique. In the actual design, the compu tation by call-method Metacircular 0 X 0 shown above is split into two pa rts. The red esigned call-method receives When running CT or RT RT (CT and) RT only method-name, looks up the targeted method, and returns a fun ction: RT: runtime, CT: compile tim e (defmethod call-method (method-name) (let ((method (<- self lookup method-name))) (lambda (object args) example of the vector library in the previous chapter. It is not sui table for (apply method (cons object args))))) dealing with adj acent but independent operations at a tim e.

If called , the function returned by call-method receives object and args and invokes the targeted method with them. The targeted method is not 3.4 Summary invoked until the returned function is called. Note that the targeted method is looked up only once when call-method runs. It is never looked up wh en This chapter illustrated currently known mechanisms for processing a pro the returned function runs. gram . We showed the tree representative mechanisms, Lisp macros, 3-Lisp, The CLOS interpreter and compiler can employ this curried protocol to and the CLOS MOP. As an object-ori ented version of 3- Lisp, we also showed improve execution performance. It should call call-method in advance at 3-KRS. load t im e or at compile time, and memoize the returned function wit h the Table 3.1 sum marizes the features of the three mechanisms. All of them method name. Then, if the method is actuall y called at runtime, it can provide meta representations of the base-level programs for the program directly call the memoized function for invoking the method. No look up mers. The meta representations ena ble the programmers to process the is needed at runtime. The targeted method is looked up only once at load programs as computable data. As the meta representations, Lisp macros time or at compil e time. provide program text in the form of the abstract syntax tree. 3-Lisp pro vides not only progra m text but also the current environment and the current Applicability to our problem continu ation. The CLOS MOP provides classes, generic functions, methods, a nd so on. The CLOS MOP shows that there is another design of the meta system of Although providing meta representations a ll ows to im plement a new kind a reflective language. Because of its metacircu larity, a meta-level program of abstraction that are not avail a ble within the confines of the original lan is not an interpreter of the base-level program but can be regarded as a guage, some kinds of abstractions also need to alter the default behavior collection of base-level code substituted for parts of the original base-level of t he language. This self modification of the la nguage is all owed on ly by program. Recall that a meta-level program describes base-level behavior 3-Lisp a nd the CLOS MOP. Lisp macros cannot change the behavior of the with the base-level primitives instead of built- in meta functions. language when readin g a variable, for exampl e. 3-Lisp a nd t he CLOS MOP, This fact means that mctacircular refl ection can be the basis of a solu tion therefore, enable a la rge r number of kind s of a bstractions than Lis p macros. of our problem. In the previous chapter, we presented that C++ needs Metacircularity is a good property because metacircular systems are easy a mecha ni sm for processing source code with context sensitivity and non to learn a nd easy to implement effi ciently. Only Lis p macros and the CLOS locality. l\ !etacircular re fl ection all ows progra mmers to specify source-code !\ lOP have this property. 3-Lisp is not metacircular; although the base level su bstitution with context sensitivity and non locality. and the meta level uses the same language, t he customization by meta func ll owcve r, we cannot usc metacircul a r refl ection represented by the CLOS tions is not a ppli ed to the langu age for the meta level. The customiza.t ion MOP as is to solve our problem. It is in principle a runtime system, which is applied only to the langu age for the base le vel.

may involve overheads, and is difficult to h;wdle optimization s uch as the Further more, macro fun ctions may run at either run time or compi le timc 1

' - ~-~-=--="'--0.------36 CHAPTER 3. TECHNIQUES FOR PROCESSING A PROGRAM whereas meta functions in 3-Lisp and metaobjects in the CLOS MOP run at runtime. This means that Lisp macros have an advantage in terms of execution performance since macro functions run at compile time. 3-Lisp has to process a program at runtime even though the processing can be done at compile time. To avoid the performance problem of 3-Lisp, the CLOS MOP employs the currying technique and performs most of meta computation at the load time or at compile time. Chapter 4 Through this chapter, we have discussed that a meta architecture with metacircularity, such as the CLOS MOP, has great benefits for processing a program. But this architecture has a drawback with respect to execution performance because metaobjects normall y run at runtime unless an elabo The OpenC++ MOP rate technique like currying is used. Lisp macros have an advantage for the performance issue; macro functions can run at compile time and they can involve no performance penalties at runtime. This feat ure of Lisp macros is significant to develop a mechanism for processing a program in C++ since This chapter presents a new C++ mechanism for processing a program. C++ programmers are particu lar about execution performance. T he proposed mechanism enables context-sensitive and non-local processing, In the next chapter, we propose a new C++ mechanism wit h both advan which is not supported by existing C++ mechanisms, such as the inheritance tages of the CLOS MOP and Lisp macros. It enables source-code processing mechanism and the template mechanism. Because of those two features, this in a context-sensitive and non-local way so that programmers can develop mechanism makes it possible to include some kinds of useful abstractions in a better libraries in C++. library and implement other abstractions efficiently. In regular C++, those abstractions are impossible to include in a library or, otherwise, they are difficu lt to implement efficiently. The proposed mechanism is called the OpenC++ MOP (Metaobject Protocol). Open C++ is an enhanced version of the C++ language for this mechanism.' The OpenC++ MOP has been developed by a synthesis of ideas of the techniques illustrated in the previous chapter. Especially, we took the basic protocol structure from the CLOS MOP [36], and we took the basic architecture from Lisp macros. The Open C++ MOP is also influenced by Intrigue [37], An ibus [50, 51], and MPC++ [33, 34].

4.1 Overview

The OpenC++ compiler is fed with two kinds of code. One is ordinary source code and the other is meta code that specifies how the source code is processed. !3oth of them are written in Open C++. The compiler first runs the C++ preprocessor, and then performs source-to-source translation from Open C++ to regular C++. This translation is specified by the meta code. The translated code is passed to the back-end C++ compiler and processed into executable code.

1To distingu ish OpenC++ version 1, lhis language is called OpcnC++ version 2.

37 38 CHAPTER 4. T HE OPENC++ MOP 4.1. OVERVIEW 39

Introductory Example The first lin e beginning with metaclass is a metaclass declaration. It de clares that the class metaobject for Point is an instance of RecordedClass. The OpenC++ MOP is a protocol that the meta code uses for specifying Thereby, the program above is translated by the class metaobject into this sou rce-code processing. The protocol structure of the OpenC++ MOP is program: based on that of the CLOS MOP. Programmers define a new clas~ metaob ject to specify a new kind of source-code processing. For example, we show class Point { t he implementation of the langu age extension that records object creation. public: int x, y; T his is the example t hat we repeatedly used in the previous chapter: };

class RecordedClass : public Class { void f(){ public : Ptree+ TranslateNew(Environrnent• env, Ptree• header, Point• p (history[n++]="Point" ,new Point); Ptree• new_op, Ptree• placement, Ptree• type_name, Ptree• arglist) } { return Ptree: :Make(" (history [n++] =\ "'l.p\", 'l.p)", Note that the metaclass declaration is eliminated a nd the new expression type_narne, "new Point" is replaced with "(history[n++]= ... )" . If the function f() Class::TranslateNew(env, header, new_op, placement, type_name, is executed, a character string "Point" is stored in a n array history. The arglist)); variable n specifies the number of the recorded class names. } }; What Class Metaobjects do

We define a new metaclass, which is a class for class metaobjects. The new The translation shown above is perform ed by TranslateNew() . It receives metaclass RecordedClass corresponds to recorded-class that the previous the program text for the new expression and the environ ment, and returns chapter s howed while mentioning the CLOS MOP. It inherits from the de the translated new expression. The environment represents the bindings faul t metaclass Class and overrides a member function 2 TranslateNew() as between names and their static types. Unlike the environment in 3- Lisp, it the cl ass recorded-class inherits from metaobject-for-class and over does not represent the dynamic bindings between names and t heir runtime rid es create-object. values. The received progra m text is represented in the form of the parse Although create-object in the CLOS MOP directly interprets the pro t ree as in Lisp macros and 3-Lisp. Ptree is the type for the parse tree. gram and executes the object creation, TranslateNew() in the OpenC++ For convenience, TranslateNew() does not receive the new ex pression as MOP just translates t he program at the source-code level as Lisp macros do. a single tree. The tree is di vided into several subtrees before passed to It receives program text and returns the res ult ing text of the translation. TranslateNew(). For example, the fift h argument type _name is bound to To use this language extension , programmers write something like this: the type-name field of the new expression, that is, Point. TranslateNew() constructs a new parse tree that is s ubstituted for the metaclass Point : RecordedClass; original new expression. To do this, TranslateNew() call s a built-in function class Point { Ptree: :Make(). Here we show the function body of TranslateNewO again: public: int x, y; return Ptree: :Make("(history[n++]=\"'l.p\",'l.p)", }; type_name, Class: :TranslateNew(env, header, new_op, void f(){ placement, type_name, arglist)); Point• p new Point; The expression Class: :TranslateNew(env , . .. ) calls the overridden mem } ber function of the base class Class. Since Class is the default metaclass, 2 this member fun ction call returns the new expression without a ny change. . A m cm l ~er fu~1ction means a method in the C++ term in ology. Similarly, in C++, an mstancc vanablc IS called a data member, and a super class is call ed a base class. The fun ction Ptree:: Make() constructs a parse tree according to the formal 40 CHAPTER 4. THE OPENC++ MOP 4.2. CONTEXT-SENSITIVE AND NON-LOCAL 41

given as the first argument. It substitutes given subtrees for the occurrences is represented by a Ptree metaobject: of /.p in the format. For example, the first occurrence of /.p surrounded by double quotes is replaced with the subtree indicated by type_narne. ((int) (a = (b + (c • 2))) ;) A class metaobjects handles all kinds of expression s involved with the Here, () denotes a lin ked list. We denote a parse tree with the notation that class. They can handle class declaration, the new expression , member func Lisp uses for the S expression. Note that operators such as = and + make tion calls, and even data member accesses. The default metaclass Class has su blists. member functions for translating each kind of expression. Although these The OpenC++ provides many functions for manipulating a parse tree. member functions of Class do not transform the received program text, Most of them were taken from Lisp. For example, to obtain the second programmers can define a new metaclass to override them and customize sublist of the list that a variable expr is bound to, the programmer writes: the translation. Note that a class metaobject handles only the expressions involved with the class. Hence programmers can restrict the customized Ptree: :Second(expr) translation to only some classes. Even though a new metaclass is defined, expressions are not translated unless the metaclass is specified for the class The function Second() is a static member function 3 of the class Ptree. It that the expressions are involved with . returns the Ptree metaobject that represents the second sublist. Moreover, since the grammar of C++ is relatively complex, the Ptree metaobjects provide a member function Whatis() for examining the kind 4.2 Context-sensitive and N on-local of the syntax represented by the parse tree. Whatis() returns a unique constant according to the kind of the syntax. If the parse tree represents a The OpenC++ MOP enables context-sensitive and non-local processing of declaration, Whatis() returns PtreeDeclarationid; if the tree represents programs, which the inheritance mechanism or the template mechanism can a class name, WhatisO returns LeafClassNarneid. not do. This feature of the Open C++ MOP makes it possible for program As we already saw, Ptree metaobjects can be constructed by calling mers to write libraries that they cannot do in regular C++. Ptree : :Make(). This static member function constructs a Ptree metaob ject according to the given format. All occurrences of I.e (character), /.d (in Context Sensitive teger), /.s (character string), and l.p (Ptree) in the format are replaced with the arguments following the format. For example, suppose that array _name The Open C++ MOP provides rich meta representation of programs so that is "xpos" and offset is 3. This function call: programmers can refer to various contextual information of the programs during the processing. For example, the programmers can refer to the pro Ptree: :Make("/.s [/.d]", array_narne , offset) gram text represented in a parse tree, the static type environment, and class definitions. The OpenC++ MOP provides different metaobjects for constructs a Ptree metaobject that represents: each kind of information. The programmers use these metaobjects and de xpos [3] termine how to process the programs. We below present brief overviews of all the kinds of metaobjects: Ptree, Unfortunately, t he current implementation ofPtree: :Make() does not con Environment, Type Info, and Class. The detailed specifications are shown struct a fully-capable Ptree metaobject. Since the C++ grammar is context in Appendix B. These metaobjects represent program text, static types, type sensitive, Ptree: :Make() cannot correctly parse the constructed metaobject definitions, class definitions, respectively. They cover information needed to without syntactic context and hence Whatis() does not work for it. Except determine the semantics of a given code fragment. this limitation, however, programmers can use the Ptree metaobject re turned by Ptree: :Make() at any place in a meta-level program. • Ptree metaobject The function Ptree: :Make() makes it significantly easy to write a meta level program . It is a conversion function from standard C++ syntax given The Ptree metaobjects represent the parse tree of a program. The parse as a C++ string to a Ptree metaobject. Thus programmers can construct a tree is implemented by a linked list of lexical tokens. For example, this Ptree metaobject with standard C++ syntax, which is more intui tive and program: easy to handle than bare new operators.

int a = b + c * 2; 3 A static member function is a sort of class method in C++.

- ---~ .~.. =~= --~~~--~~-"«••··------~ 42 CHAPTER 4. THE OPENC++ MOP 4.2. CONTEXT-SENSITIVE AND NON-LOCAL 43

The Open C++ MOP also provides a function for pattern matching. The metaobject for int. If the type is a function type, Dereference() static member function Ptree: :Match() compares the given pattern and returns the return type of the function. the given Ptree object. If they match, it returns TRUE and binds the given variables to the appropriate sublists. See the following sample program: BOOL NthArgurnent(int nth, Typeinfo& t) This works for function types. It returns the type of the nth argument if (Ptree: :Match(expr, "['l.? + 'l.?]", &lexpr, &rexpr)) to the function. If there is not the nth argument, NthArgurnentO cout <<"this is addition.\n 11 ; returns FALSE. else if(Ptree::Match(expr, "['l.?- 'l.?]", &lexpr, &rexpr)) cout << 11 th is is subtraction. \n••; else o Class metaobject cout << "unknown\n"; The Class metaobjects supply member functions for introspection. Pro The pattern ['l.? + 'l.?] matches a Ptree metaobject if the length of the grammers can call these functions to i~spect the class; for example, they linked li st is three and the second element is+. If expr matches the pattern, can obtain the class metaobject for the base class. The followings are part lexpr gets bound to the first element of expr, and rexpr gets bound to the of these member functions: third element. Note that the type of lexpr and rexpr is Ptree. Ptree• Name() o Environment metaobject This returns the name of the class.

The Environment metaobjects represent bindings between names and their char• MetaclassName() static types. Programmers can call Lookup() for the metaobjects to deter This returns the name of the metaclass. mine the type of a variable name. The returned type is represented by a Typeinfo metaobject. Class• NthBaseClass(Environment• env, int nth) This returns the nth base class of the class. Recall that C++ allows multiple inheritance. The leftmost base class is the first, and the o Typeinfo metaobject rightmost is the last. Note that this member function returns only The Typeinfo metaobjects represent types. The types are not limited to immediate base classes. It does not return a base class of a base class. class types. They also include other kinds of types, such as built-in types like int, pointer types, function types, template types, and so on. Ptree• NthMemberName(int nth) The most important member function of Type Info is \lhatis () . It re This returns the name of the nth member, which is either a member turns a unique constant according to the kind of the type. For example, if function or a data member. the type is a class type, llhatis() returns ClassType. BOOL LookupMemberType(Environment• env, Ptree• name, The Typeinfo metaobjects also supply member functions for obtaining Typeinfo& t) detailed information of each kind of type: This returns the type of the member specified by name. uint IsBuiltinType() This works for built-in types. It returns what the built-in type is, Non Local char, int, double, or others. The OpenC++ MOP also enables non-local processing. l3ecause a single Class• ClassMetaobject() class metaobject processes a ll the code fragments relevant to the class, pro grammers usually define only one new metaclass for implementing one ab This works for class types. It returns the Class rnetaobject for the class. straction . Even if the processed fragaments spread out over the whole pro gram, they are automatic

For non-local processing, the Class metaobjects s upply a different mem • Type Modifier ber function for each kind of program text. One of these fun ctions is Programmers can define a new type modifier. It may appear in front TranslateNew() presented in Section 4.1. T hese functions receive program of type names, the new operator, or class declarations. For example, text and translate it. The t ranslated text is substituted for t he original programmers may define new keywords, distributed and remote, text in the program. Although t he member fun ctions supplied by Class do and then t hey can write: not change the received text at all , a subclass of Class ca n override them and implement new source-code translation. The followings are some of the distributed class Dictionary { ... }; member functions that the subclass can override (For the complete list , see Appendix B): remote Point• p = remote(athos) new Point;

- Ptree• TranslateClassName(Environment• env, Here, distributed, remote, and remote(athos) are new type modi Ptree• keyword, Ptree• name) fiers. This translates the class name appearin g in the program. • Access Specifier Ptree• TranslateSelf(Environment• env) Programmers can define a new member-access specifi er, which appears T hi s translates the cl ass declaration. within class declarations. public, protected, and private a re the built-in access specifiers. For example, if after is a keyword for a new Ptree• TranslateMemberFunctionBody(Environment• env, access specifi er, then programmers may write: Ptree• name, BDD L inlined, Ptree• body) This translates the body of a member function supplied by the class. class Window { public: Ptree• TranslateUnary(Environment• env, Ptree• op, void Move() ; Ptree• object) after: This translates a n expression includin g a unary operator op. It is void Move() { ... } II after method call ed if the object that the operator is appli ed to is a n instance of }; t he cl ass.

Ptree• TranslateNew(Environmento env, Ptree• ... ) • Statement This translates the new expression. Programmers can defi ne a new kind of statement that is simil ar to ei ther the if statement or the while statement. The foll owin g examples Ptree• TranslateMemberRead(Environment• env , Ptree• member) are valid syntax extension s: This translates an expression for readin g a data member of an object of the class. Matrix ml; Ptree• TranslateMemberCall(Environment• env, Ptree• member, ml.forall(e){ e 0 . 0; } II extended syntax Ptree* argl ist) This translates a member function call on an object of the class. ButtonWidget b; b.press(int x, int y){ II extended syntax cout << 11 pressed at•• << x << •• •• << y; 4.3 Syntax Extension };

The OpenC++ MOP allows limited syntax extension. Because the C++ gram mar is heavily context dependent, the full extensibil ity fo r syntax is In these examples, forall and press a re new keywords. Like the diffi cult to provide in C++. However, programmers can define new keywords while statement, they arc followed by a () expression and a block and implement the follow ing kind s of new sy ntax: statement. 46 CHAPTER 4. THE OPENC++ MOP 4.4. WHAT IS NEW? 47

To make these syntax extensions available, the programmers first register Comparison with Lisp Macros the keywords to the parser. The Class metaobjects supply the following registration functions. Each one registers a keyword for a specific syntax In the Open C++ MOP, class metaobjects process a program as Lisp macros extension (the followings are not all the functions. See Appendix B for do. Their mem bcr functions receive program text and return the translated more details): text, which is s ubstituted for the original text. However, the Open C++ provides more contextual information than Lisp - void RegisterNewModifier(char• keyword) macros when translating a program . The member functions for the trans This registers a keyword for a new type modifier. lation receive the environment as well as program text. They can use it to examine the static types of variables. Lisp macros do not provide the envi - void RegisterNewAccessSpecifier(char• keyword) ronment; macro functions have to translate a program only with syntactical This registers a keyword for a new access specifier. information but without contextual information. void RegisterNewWhileStatement(char• keyword) The Open C++ MOP also enables self modification whereas Lisp macros This registers a keyword for a new kind of statement that is similar to do not. With only a simple annotation, it can alter the behavior of the the while statement. objects of only a s pecific class. Error-prone programming conventions are not needed. Lisp macros require programmers to explicitly call them to The semantics of a new syntax extension is defined by the class metaob translate programs. The expressions that a Lisp macro should process have ject that is involved with the extension. For exampl e, the forall statement to be expli ci tly preceded by the macro name. Hence, if programmers want in this program: to change the behavior of the language when reading a variable, they have to write something like (read-variable x) in stead of just writing x. Here Matrix ml; read-variable is a macro name. Otherwise, programmers have to write the ml.forall(e){ e = 0.0; } code walker. On the other hand, the Open C++ MOP does not need such a Should be translated into regular C++ code by the class metaobject for programming convention. Once a metaclass is declared, all the expressions Matrix. The programmer, therefore, has to define a new class metaobject to involved with the class are automatically processed by the class metaobject. handle the translation . The new class metaobject will override this member Programmers can alter the behavior of the objects without editing the origi function: nal program to in sert something like macro nam es . Moreover, programmers can restrict the range of the self modification within a specific class. The - Ptree* TranslateUserStatement(Environrnent* env, behavior for the other classes are kept as is. Ptree* object, Ptree* op, Ptree• keyword, Ptree• rest) Comparison with the CLOS MOP This translates a new kind of statement. The default implementation by Class causes a syntax error. The CLOS MOP is t he immediate a ncestor of the OpenC++ MOP. Both the CLOS MOP and the Open C++ MOP are metacircular, and they employ If the forall statement is translated , the a rguments object, op, and keyword class metaobjects instead of the metaobjects for objects. The difference is are bound to ml , . (dot), and forall , respectively. The last argument rest that the OpenC++ MOP is special ly designed to run at compi le time. is bound to the rest of the statement, that is, (e){ e = 0.0; } . The over Although the currying technique allows metaobjects in the CLOS MOP ridden TranslateUserStatement() should use these a rguments and con to mostly run at co mpile time, some computation by the metaobjects is still struct the parse tree for the translated statement. performed at runtime. At least, which metaobjcct is selected for given base level code is determined at runtime. See Figure 3.3 in page 32 again. The 4.4 What is New? default interpreter (or compiler) has to determine whether it executes each expression through a mctaobjcct or not, and which metaobject it selects if T he Open C++ MOP has been developed by a synthesis a nd re-engineering so. In the CLOS /\ LOP, this is done at runti me. of ideas of other known techniques. Especially Lisp macros and the CLOS The OpcnC++ performs a ll meta computation at compile time. It also MOP influ ence the design of the OpenC++ MOP. This section discusses determines which meta.object is selected at compile time. 13cca.usc of this comparison between the OpenC++ MOP and other techniques. feature and the metacircularity, the OpenC++ MOP docs not imply any 48 CHAPTER 4. THE OPENC++ MOP 4.5. SUMMARY 49

performance penalty at runtime. The Open C++ MOP uses static typin g to Moreover, Anibus's program translation is to transform a parse tree. The select a metaobject at compile time. It statically types all variables and ex metaobjects perform the transformation in stead of generating substituted pressions in the program and determines which metaobject is responsible for source code. the translation . Then it calls the metaobject for translating the expression CRML [31] has a compile-time MOP for ML (43]. Its MOP provides the and directly substitutes the result for the origin al expression. This means capability similar to Lisp macros and makes it possible to use new syntax that even the code for selecting a metaobject and dispatching to it does not in ML. However, it also involves the same li mitations as Lisp macros. For run at runtime. example, it does not enable self modification. The Open C++ MOP is regarded as a good synthesis of the C LOS MOP The MPC++ MOP (33, 34] is another compile-time MOP for C++. a nd Lisp macros. We below present the definition of RecordedClass written As in OpenC++, it allows programmers to control source-code translation. in pseudo Lisp. Comparing this defin ition with the equivalent defi nition in The most significant difference between MPC++ and Open C++ is that the the CLOS MOP, the readers will intuitively understand the synthesis. First, MPC++ MOP does not provide non-locality based on the class system. we show the definition in the Open C++ MOP: Instead, each piece of program is dispatched to a metaobject for transla tion according to the kind of the parse tree, such as a declaration, an if (defclass recorded-class (metaobject-for-class) (defmethod create-object (env expr) statement, a + operator, and so forth. Metaobjects of MPC++ are nodes (let ((class-name (<- self name))) of the parse tree of the processed program. If a piece of program is dec '(begin (set! •history• (cons ,class-name •history•)) laration of a variable, for example, then it is dispatched to the declaration ,(<-super create-object env expr))))) metaobject {corresponding to t he Ptree metaobject in OpenC++) for the translation, rather than the class metaobject involved with the type of the The next is t he definition in the CLOS MOP: declared variable. (defclass recorded-class (metaobject-for-class) Because of this feature, the MPC++ MOP is not suitable for customiza (defmethod create-object (init-args) tion that is specific to a particular class and in volves several statements and (let ((class-name (<- self name))) expressions. If the customization needs to translate declaration statements (set ! •history• (cons class-name •history•)) a nd -> expressions on a specifi c class, t hen the programmer needs to de ( <- super create-object init-args)))) fine two new metaclasses for declaration a nd the -> operator. Then the The readers can see that the two defin itions are significantly close to each new metaclasses must explicitly determine the class of the processed code other. The primary difference is t hat the defi nition in the OpenC++ MOP and translate t he code only if it is the specifi c class. This computation is retu rn s an expression instead of directly executing the expression. T his implicitly performed in OpenC++ because it provides non locality. feature is t he influ ence by Lisp macros. Moreover, the MPC++ MOP does not provide the meta- meta level. Thus programmers cannot write a meta-meta level program to make it easier Comparison with Early Compile-time MOPs to write a meta-level program . Since the Open C++ MOP provides the meta meta level, for example, programmers can enjoy special syntax when wri t in g Several MO Ps running at co mpile time, call ed co mpile-time MOPs, have a meta-level program. been developed earlier t ha n the OpenC++ MOP. The earli est compile-time 10Ps are Intrigue (37] and Anibus [50, 51]. They are designed for Scheme (16] and parallel Scheme although they are written in CLOS. However, their 4 .5 Summary design is quite different from the Open C++ MO P. They provide meta in ter face mainly for customizing the behavior of t he compiler rather than source This chapter proposed the OpenC++ MOP, wh ich is a mechanism for pro code t ranslation . In fact, metaobjects of In trigue are intern al components of cessing a program. This mechanism provides rich meta representation of t he compiler, such as a parser a nd an optimizer. IVIetaobjects of Ani bus a re the processed program and enables co ntext-sensitive and non-local pro nodes of a parse tree and co ntrols tra nslation from pa rallel Sc heme to reg cessing. for context-sensitive processing, the OpenC++ MOP provides ular Scheme including low-level primitives for parallel computing. Ani bu s's Ptree mctaobjects (for program text) , Environment metaobjects, Typeinfo MOP might look simil ar to the OpenC++ MO P since bo th MOPs co ntrol mctaobjects, and Class metaobjects. These metaobjects are t he meta rep program translation , bu t it docs not provid e non-locality based on the class resentation of various aspects of the program. The Class mctaobjects also system (or the type system because Scheme does not include a class system). work for non-local processin g. They control t he t ranslation of a ll the ex- 50 CHAPTER 4. THE OPENC++ MOP

pressions that are spread out over the whole program but involved with the class. Like Lisp macros, the OpenC++ MOP also enables syntax extensions. Programmers can register a new keyword and defin e a new type modifier, an member-access specifier, or a new kind of statement. They are appropriately tra nslated into regular C++ code by Class metaobjects as other code is translated. Chapter 5 These features are advantages of the Open C++ MO P against other ex isting C++ mechanisms such as inherita nce a nd templates, and they enable C++ libraries that have been impossible or difficult to make them efficient. Meta Helix To enable those libraries, a n extended C++ compiler specially prepared for the libraries has been necessary so far. The OpenC++ MOP all ows pro gram mers to write a meta-level program to implement such an extension to C++ on top of the compiler. Like the CLOS MOP, the OpenC++ MOP is metacircular since metacir The OpenC++ MOP has been developed by a synthesis of ideas from cular systems are easy to learn and make it easier to write efficient meta Lisp macros and the CLOS MOP show n in the previous chapter. However, programs. Unfortunately, pure metacircul arity can lead to a problem we it is not just a C++ version of Lisp macros or the CLOS MOP. It is some call implementation level con fl ation. This problem confuses programm ers thing considerably different from Lisp-style macros or traditional reflective and causes errors, hence we could not adopt pure metacircul arity as is for languages. the OpenC++ MOP. The basic architecture of the OpenC++ MO P was taken from Lisp Instead, to avoid the problem, we have developed an improved version of macros. In both of them, the meta-level program receives program text metacircular a rchitecture, named the meta helix [15]. This chapter presents and returns the translated on e. Moreover, th e meta-level program can run a problem of the pure metacircular a rch itectu re, and proposes the meta heli x at com pile time to improve execution performance of the base-level program. as a solution. The meta helix preserves advantages of metacircu larity but On the other hand, the Open C++ MOP provides ri cher meta representation also addresses the problem we present. than Lisp macros. It provides not only program text but also a n environ ment and type information . Also, it provides non-locality based on the class system of C++. Furthermore, self modification is possible in the Open C++ 5.1 Implementation Level Confl.ation MOP. Programmers can alter t he behavior of the language without pro gramming conventions or th e code walker. In a metacircular system like the CLOS MOP, the language can be cus The protocol structure of the OpenC++ MOP is based on that of the tomized in t he customized language itself. This means not onl y that the CLOS MOP. The OpenC++ MOP is metacircu la r, and class metaobjects base-level program a nd t he meta-level program are wri tten in the same lan a re the subjects of processing a program. Howeve r, the OpenC++ MOP guage, but also that th e customization by the meta-level program refl ects on is design ed so t hat metaobjects run at co mpile time. All the meta co mpu the language in which t he meta-level program itself is written. This fea ture tation, that is, source-code processing, is performed at compile time. Even gives advantages t hat we presented in Chapter 3, but it can also lead to a which metaobject should be selected for processing is determined at com problem wh ich we call implementation level conjlation. pil e time. The CLOS MOP perform s this selection at runtime although other meta computation can be moved to co mpile time with the currying Lisp macros technique. First of all, we show an example of implementation level co nfl ation in Lisp macros, which is also metacircular. Sin ce Lisp macros are relatively sim pler t han the CLOS ~ lO P, this example might look trivial. However, we be lieve t hat showing t hi s example help t he readers und erstand a more seriou s exam pie we show later. Suppose that we mod ify the behavior of a special form define so that

51 52 CHAPTER 5. META HELIX 5.1. IMPLEMENTATION LEVEL CONFLATION 53

more than one variables can be defined at the same time. With this exten A More Serious Example sion, for example, we can define variables x and y with the initial value 7 by The problem of the example shown above is that two distinct concepts, this single expression: which are the define special form and the define macro, a re conflated into (define x y 7) a single name. We call this problem implementation level conflation because the implemented level (define macro) and the implementing level (define This expression should be expanded by a macro function named define in to special form) are confl ated into a single structure. this:' This conflation becomes more serious in metacircular systems like the (begin (define x 7) (define y 7)) CLOS MOP. The power of the CLOS MOP is that programmers can enjoy metacircularity and use the existin g facilities of CLOS as much as possible Therefore, the definition of the macro should be this: when they write meta-level programs. But this is a double-edged blade. It can also easily lead to conflation of a new abstraction and the facilities (define-macro (define . args) implementing that abstraction. (let ((vars (list-head args (- (length args) 1))) (init (car (last args)))) To illustrate more serious implementation level conflation, we show an '(begin .~(map vars example of an extension written with the CLOS MOP. This extension records (lambda (v) '(define ,v ,init))) ))) all accesses to slots (i.e. data members) of objects. The following program is an example of a program that uses this extension:2 Unfortunately, this macro does not wo rk; it will fall down in to an infinite loop. Because the symbol define is now a macro name, occurrences of > (defclass point () define in the expanded expression are repeatedly processed by the macro (variable x y) :meta history-class) function. For example, the expanded expression: POINT > (define p1 (make-instance point)) (begin (define x 7) (define y 7)) P1 > ( <- p 1 x 3) set x to 3 3 Will be expanded again into: > ( <- p1 y 2) set y to 2 2 (begin (begin (define x 7)) (begin (define y 7))) > ( <- p1 x) read x 3 And this expression also invokes macro expansion, and so on. > (slot-history p1) show the access log This problem can be fixed although the way of fixing is quite imple ((GET X) (SET Y) (SET X)) mentation dependent. To fix this problem, we have to define the macro as follows: Since the metaclass for point is history-class, all the accesses to t he slots of point objects a re reco rd ed, and the access log is available through the (define define• define) I I alias function slot-history. (define-macro (define . args) (let ((vars (list-head args (- (length args) 1))) Slots with access hi story can be implemented usi ng the existing slot (init (car (last args)))) mechanism. The way this works is simply for instances of point to actually '(begin .~(map vars have three slots, the two visible slots x and y as well as a third, "hidden" (lambda (v) slot history for storing the access log. The definition of history-class '(define• ,v ,init)))))) II use define• implemented in this way is as follows: This program first binds a symbol define• to the original define special (defclass history-class (metaobject-for-class) form. i\ow define• is an alias of define. Then, the program defines a ; what slots do the instances have? macro define and uses define• for the expand ed expression. Us in g the (defmethod compute-slots () alias makes the macro function avoid the infinite loop since the expanded (cons 'history (<- super compute- slots))) ex press ion does not include the macro nam e define any more. ; read a slot of an in stance 1 Por si mplicity, we assume that the sub-expression for the initial value docs not cause 2 side-effects. Again , we use the same altered syntax that we used in Chapter 3. 54 CHAPTER 5. META HELIX 5.1. IMPLEMENTATION LEVEL CONFLATION 55

(defmethod read-slot (object slot-name) Problems for the Implementor of the Extension (< - object history ' ((get ,slot-name) ,10(<- object history))) Implementation level confiation also can cause problems for the im plementor (<- super read-slot obj ect slot-name)) of the extension. A careful reading of the method read-slot supplied by (defmethod set-slot ... )) history-class provides an example of this. The im plementation of read-slot has a bug which manifests itself as in finite recursion although it is better understood as resulting from the im This metaclass overrides three methods. First, it overrides compute-slot plementation level conflation . Operationally, the bug is that the body of so that every history class has an extra slot to store the access log. Also the method, as part of updating the access log, must read the history slot, it overrides r ead-slot and set-slot, which reads and wri tes a slot of the which runs this method recu rsively, which starts to update the access log, instance of history classes. They updates the access log before actually wh ich reads the history slot, and so on ad infinitum. This bug happens reading or writing. Since the access log is stored in the hidden slot history, because read-slot is im plemented with a non-extended slot history but they read and write the slot to update the access log. read-slot alters the language to record all the accesses to the slots, includ Unfortunately, this example includes implementation level conftation . ing history. There are two distinct but confl ated concepts in this implementation. One The stand ard solu tion to th is problem is to introduce a special purpose is a point object t hat has extend ed slots. The accesses to the slots are test that prevents the infinite rec ursio n. So the revised code ends up looking recorded in the access log. The other concept is a point object th at has something like this: three non-extended slots, x, y, and hidden history. This object is used to implement the former object. Since t he CLOS MOP is metacircular, the two (defmethod read-slot (object slot-name) point objects are identical - the same object. Again we are dealing with (unless (eq slot-name 'history) the im plemented level (extended slots) and the implementing level (non (< - object history extended slots) but the two levels are conflated into a single structure (po int '((get ,slot-name) ,10(<- object history)))) (< - super read-slot object slot-name)) object).

This solution, while effective, seems ad hoc, can be difficult to reason about, Problems for Users of the Extension and is not effective in general.

Implementation le vel con fl ation res ults in confusion for users of the exten sion. For example, co nsider that a programmer uses one of the MOP's The Importance of an N-to-N Correspondence introspective facili ties to ask what slots a class has: The problems show n above can be better described as having to do with implementation level co nflation. As mentioned at the beginning, there are > (class-slots (find-class 'point)) two concepts of slots in pl ay : extend ed slots x a nd y, the accesses to which (HISTORY X Y) are recorded, and non-extended slots x, y, and history, which are used to implement the extended slots. But, because of the conflation, there is on ly According to the specification s of the C LOS MOP, class-s lots returns the one structure, or a handle to refer to the two concepts. The bug happens li st of the slot names that the given class has. The resu lt, ho\vever, in cl ud es si nce the ha ndle always refers to the extended slots. If, somehow, the slots the history slot. Is this right? Wh at does "the give n class" mean in the within the body of r ead- slot cou ld refer to non-extended slots, then t he specifications? Is it the implementing class or t he implemented cl ass? It ad-hoc solu tion to the infinite recursion could be avoided . s hould be t he implemented class and hence, since the history slot exists Fundamentally, if there are n distinct important views of an object- or only for the implementation, including the slot is entirely in a ppropriate. any other structure - t here needs to ben distinct ha ndles to it. For exam This problem particu la rl y s hows up when usin g browsers and debuggers ple, we need some way for in stances of the class point to be viewed in terms that rely on the introspective part of the CLOS MOP to work. Exposing of either the implemented functionality or the implementing functionality, this detail of how slots with history is implemented can leave progra mm ers not a conflation of both. The users of the extension want a view in terms of co nfused, or worse yet, can tem pt t hem to rely on this im plementation details the implemented functionality, bu t the implementor of t he extension wants in ways that they should not. to be able to take one view or the other at different times. 56 CHAPTER 5. META HELIX 5.2. INADEQUATE SOLUTIONS 57

5.2 Inadequate Solutions Tiny CLOS MOP

Implementation level conflation is a flaw of the metacircular architecture. If we give up metacircularity, we can avoid implementation level confla Despite the advantages, this architecture makes metaobject protocols lose tion. Non-metacircular systems such as 3-Lisp and 3-KR.S do not involve elegance; the metaobject protocols become inconsistent and difficult to use. this problem because they have two distinct handles for implemented func Because of this fact, we could not adopt the pure metacircular architecture tionality and implementing functionality. In 3-KRS, if programmers want for the OpenC++ MOP. Instead, we have developed a improved metacir a view in terms of the implemented functionality, they s hould refer to the cular architecture since the OpenC++ MOP should be elegant, simple, and base-level objects, but if they want a view in terms of the implementing easy to use, as much as possible. functionality, they should refer to the associated metaobjects. Since there Before presenting our solution to the problem of implementation level is no metacircularity, the base-level objects and the metaobjects are never conflation, we first present two earlier solutions that, in different ways, fail conflated . The behavior of the metaobjects does not change depending on to our needs. These solutions serve to further flesh out the criteria which the metaobjects themselves. the more general solution should meet. The Tiny CLOS MOP developed by Gregor Kiczales et al avoids imple mentation level conftation by partially giving up metacircularity. The Tiny CLOS MOP provides two different abstractions for per instance storage: Change the Implementation slots and fields. Fields are a lower-level abstraction used to implement slots; One possible solution to this problem invol ves implementing the extension they represent memory image allocated for implementing each object. The in a different way, specificall y by storing the access log in the class metaob base-level Tiny CLOS programs never know the fields exist. ject rather than directly in the objects themselves. The followin g program In the specific example of the history class, the extension works bi al implements history classes in this way: locating an extra field for each object. So, for example, point objects have two slots x andy; but they have three fields, for holding the x and y slots and (defclass history- class (metaobject-for-class) the slot access log. The implementation of history-class in Tiny CLOS (variable history) ; place to store the access log looks like: (defmethod read-slot (object slot-name) (defclass history-class (metaobject-for-class) (let ((log (assq object (<- self history)))) (variable history-index) the ind ex of the field that will (set-cdr! log '(get ,slot-name) .~(cdr log)) (<- super read-slot object slot-name))) ; store the access log for instances ; of the class (defmethod set-slot ... )) ; allocate an extra field and remember its index (defmethod compute-fields () All the access logs for the in stances are stored in the slot history of the (<- self history-index (<- self allocate-field)) class metaobjects (that is, a class variable in the Small talk terminology). So (<- super compute-fields)) the value of history is a li st of pairs of an object and its access log. This (defmethod read-slot (object slot- name) association list can be searched by assq with using object as a search key. (let• ((index (<- self history- index))) This solution solves the specific problems mentioned in the previous sec (<- self set-field object index tion , but it loses advantages of metacircularity, which we would like to keep. '((get ,slot-name) .~(<-self get-field object index)))) This solution must manually implement the mapping from individual objects (<- super read-slot object slot-name)) to their access logs , even though that basic functionality is already present in CLOS. Implementing that mapping is not needed if each access log is di (defmethod set-slot . . . ) ) rectly stored in a slot of the object. This solution is not only redundant, but also difficult to achieve sufficient execution performance. The implementor Fields are accessed through the methods get-field and set-field, and of history-class has to implement as efficient a mapping mechanism as each field· is specified by its index instead of its name. Fields have a more the default one, which is a slot of an object. At least, the implementation primitive naming mechanism in terms of indices. shown above does not satisfy this performance criterion; the association li st Again, this solution so lves the specific problems mentioned in the pre is significantly inefficient. vious sectio n. Implementation level conflation is avoided because (1) no 58 CHAPTER 5. META HELIX 5.3. THE META HELIX ARCHITECTURE 59

"hidden"slot for the accesses log is added, and (2) the access log stored use of the same a bstraction for the two handles. For exam pie, in the case of in an object is retrieved t hrough the lower-level methods get-field a nd the his tory- class extension, our solution has two class metaobjects, point set-field, which do not invo ke read-s lot rec ursively. Since the Tiny and point•, to represent di ffe rent implementation levels (Figure 5.1). T he CLOS MOP provid es two distinct handles, slots and field s, for implemented class point correspond s to the class in t he extended language, which keeps fun ctionali ty a nd im plement in g fun ctionali ty, programmers can select a n slot access histories, wh il e t he class point* corresponds to t he class in the appropriate view by switching the handles. non-extend ed language, wh ich is used to implement t he extend ed one. Unfortunately, this solu tion has signifi cant problems of its ow n whi ch Note t hat the two handles indicate the same entity. For example, the make it un sui table as a general solu tion. First, t his solution loses ad van class poi nt and the cl ass point• corresponds to the same class. T hey tages of metacirc ul arity. Because t he Ti ny C LOS MOP is not metacircul a r are just han dles to distin ct views of the same entity as slots and field s are in term s of slots, progra mmers have to learn lowe r-level abstractions and hand les to distinct views of per-instance storage in the T in y CLOS MO P. wri t ing effi cient meta-level programs is diffi cult. Second, t his solution is point is the handle to the implemented view a nd point • is the ha ndle to only effective in the presence of a single extension to slot fun ctionality. If, t he implement ing view . for example, someone wa nted a n additional extension (i.e. to store t he o l:> Although the two handles, such as point and point•, a re often ve ry sim jects in a persistent database [47]) there wo uld still be conflation. T his is ilar, the relationship between t he handles does not fi t the usual s ubclass-of because in such a situation there needs to be (at least) three views. The relationship. It cannot be either the instance-of or base-meta relation view of persistent objects with a slot access history, built on top of the ships. To capture the relationship between t he two handles, we introduce a n view of persistent objects, built on top of ordinary objects. But the Tiny implemented-by relationship. In Figure 5.1, the class point is implemented CLOS MOP provides only two levels, so there will still be some confl ation. by t he cl ass point•, and the object pl is implemented-by t he object pl*. To avoid implementation level conft ation in n levels of implementation, the The objects pl a nd pl* are in sta nces-of point and point•, respectively. MOP must provide support for n different views. If the im plemented-by relationship is used , the pro blems by implementa tion level co nfl ation can be easil y solved . First , the programmer who want 5.3 The Meta Helix Architecture to ask wh at slo ts a cl ass has can obtain an appropriate res ul t: > (class-slots (find-class 'point)) T he com mon idea underlying t he two unsatisfactory solutions is to distin (X Y) g ui s h the implemented and implementing fun ctionality by using a different > (class-slots (implemented-by (find-class 'point))) "ha ndle" for each. In the first proposed solution, the objects are the ha ndle (HISTORY X Y) to t he implemented fun ctionality, a nd the cl ass metaobject is the handl e to the implementing fun ctionality. In the second proposed solution , slots are Second , the method read-slot becomes more simple and easy to read . No the ha ndle to the implemented fun ctionality, while field s are the hand le to ad-hoc techniques for avoidin g infinite rec ursion a re needed any more: the implementing functionality. (defmethod read-slot (object slot-name) Bu t the problem with both of these solu tions is that the benefits of (let ((object• (implemented-by object))) metacircul arity is destroyed by the fact that the two ha ndles are distinct (<- object• history a bstractions. The id ea of our proposed solution is to address its problem by '((get ,slot-name) .~(<- object• history))) (<- super read-slot object slot-name))) prov idin g two ha ndles, but to retain t he benefi ts of metacircula rity by the Note that read-slot is t he method for not t he cl ass metaobject for point• p1• - point• but t he class metaobject for point. The behav ior of the object pl, whi ch ~ ~ (x y history) has slots wi th history, is co ntroll ed by t he class metaobject for point. Our choice of t he na me meta-heli x for t hi s arc hi tecture is best seen when p1 - point ____,.. instan ce-of thinking in term s of the relation between hand les t hat the diffe rent solu (x y) - ~ impl emented-by tions use. As s how n in Figure 5.2, in the pure metacircul ar approach, the implementation loops directl y back onto itsel f- leading to confl ation. In Figure 5.1: Two handles with the same a bstraction the Tiny CLOS approach, the implementation ma ps betwee n two distinct function alities-leading to added complexi ty. In the meta-heli cal approach, 60 CHAPTER 5. META HELIX 5.4. IMPLEMENTING THE META HELIX 61

plain slots the implemented-by relationship. The way of realizing the implemented-by fields relationship is a main issue for implementing the meta helix architecture. &slots Then this section shows how the implemented-by relationship should be rei slots slots ~tory tied for the CLOS MOP. Through this example, we present that the meta helix architecture is appli cable to not only the OpenC++ MOP but also Metacircular TinyCLOS Meta Helix other kinds of MOPs.

Figure 5.2: The implementation relation between interfaces The OpenC++ MOP is Meta-Helical In the Open C++ MOP, the implemented and implementing levels are natu rally separated; the implemented level is the original program before trans the implementation spirals between two distinct handles of (nearly) identi lation and the implementing level is the resulting program of the translation. cal functionality-preserving what is good about the metacircular approach, To implement the meta helix architecture, therefore, we should be just able while still reifying the distinction that prevents conflation of implementation to distinguish classes in the original program from the classes in the result levels. ing program, and relate the corresponding classes by the implemented-by Similarly, the meta hel ix works when there are more than two imple relationship. mentation levels. So, for example, in the case of the persistent history class The implemented-by relationship in the OpenC++ MOP is reified with mentioned in the previous section, we can create the three levels that are an aliasing technique, which we used to avoid implementation level con fl ation needed to maintain separate views, and relate them using implemented-by caused by the macro define in Section 5.1. To do this, the Open C++ MOP relationships. This is shown in Figure 5.3, which illustrates the heli cal na includes the followin g rule to be meta-helical: ture of this architecture. While distinguishing implementation levels, the meta helix archi tecture • Any class appearing in the original program must be renamed in the preserves the benefits of metacircularity because the meta helix architecture resulting program of the translation . is a super set of the metacircular architecture. Except the implemented-by relationship, the meta helix architecture is quite identical to the metacircular This rule relies the implemented-by relationship. For example, in the exam a rchi tecture. So, for example, unlike the Tiny CLOS MOP, programmers do ple of history classes, the class point shou ld be renamed point•. This alias not need to learn new abstractions such as fields to implement an extend ed ing gives programmers two handles to the implemented and implementing concept of slots. Also, writing an efficient meta-level program is still easy. functionali ty. One handle is the class point for the implemented function ality, and the other is the class point• for the implementing fun ctionality. plain slots By switching the two class names, programmers can avoid implementation . history slots level conflation. first, t hey are not confused by the introspective part of the persistent MOP any more:3 history slots > (class-of 'point) (X Y) A Three level Meta Helix > (class-of 'point•) (X Y HISTORY) Figure 5.3: The Meta l-l elix supports n implementation levels. Also, the method read-slot is implemented without confusion in an intu itive way (This implementation uses nested backquotes for emphasizing the si mil arity to other versions of implementations. Although nested backquotes might make the program look complex, the complexity does not result from 5.4 Implementing t he Meta H elix the meta helix.):

The OpenC++ MOP is based on the meta helix architecture. This sec 3 Por making: Lhc arg:umenL clearer, we usc Lisp·style syntax to show an OpenC++ tion first shows this fact, especially focu sing on how the OpenC++ reifies program. 62 CHAPTER 5. META HELIX 5.5. SUMMARY 63

(defmethod read-slot (env object slot-name) As in the OpenC++ MOP, the class point* is the handle to the imple '(begin(<- ,object history menting functionality. The class metaobject for point delegates most of its '((get ,',slot-name) ,0(<- ,object history))) work to the class metaobject for point•. For example, when an in stance of ,(<-super read-slot env object slot-name))) point is created, the class point asks the class point• to create a point• For example, this method translates an expression: object. The primitive function implemented-by on a point object returns the point• object. The slot access primitive read-slot is then specialized (<- pl x) by the class metaobject for point. Its implementation is:

into this expression: (defmethod read-slot (object slot-name) (let ((object• (implemented-by object))) (<- object• history (begin (<- pl history '((get ,slot-name) ,0(<- object• history))) '((get x) ,0(<- pl history))) (<- super read-slot object slot-name))) (<- pl x))

Note that the type ofpl is not the class point but the class point* after the The method read-slot first gets the implementing object object• for that translation. During the translation, all occurrences of the class name point object and then updates its history slot. The method finally invokes are replaced with point•. Therefore, after the translation, the expression read-slot suppli ed by the su per class, which delegates the actual work ( <- pl history) is not recursively processed by the class metaobject for of implementation to the class metaobject for point•. point. Rather, it is processed by the class metaobject for point•, which is a distinct metaobject and would be the default one. If programmers want 5.5 Summary to recursively process, they can declare that the type of pl is still the class point. This chapter presented a new analysis of a problem that a rises in existing The Open C++ MOP also supports multiple implementation levels. The metacircular systems. This analysis shows how pure metacircularity causes persistent history slots shown in Figure 5.3 can be implemented by just confusion when there are not clearly distinguished views of the implemented specifying a non-default metaclass persistent-class for point+. If the and implementing functionality. We call this problem implementation level metaclass is not the default one, all the expressions involved with the class conflation. This chapter first shows an example of the confusion in Lisp point• are recursively translated by the class metaobject for point+. This macros1 which is a simple metacircular system. Then it shows a more serious process is repeated until the metaclass becomes the default one. example with the CLOS MOP. The confusion makes troubles for both the users of the language extension a nd the implementor of the extension . A Meta-Helical Version of the CLOS MOP Because of implementation level conflation, we did not adopt the pure metacircular architecture for the OpenC++ MOP. Although implementa The meta helix is applicable to other kinds MOPs such as the CLOS MOP. In tion level confl ation does not reduce the capability of the MOP for process the OpenC++ MOP, the implemented and implementing function ality are ing programs, it severely impacts the elegance of the design of the MOP. The naturally separated into the original program and the translated program. In elegance is a sign ifi cant matter because making it really easy for progarm the CLOS lv!OP, however, the implemented and implementing functionality mers to prod cess programs is one of the design goals of metaobject protocols coexist in the same runtime environment. So we need a different technique in general. to reify the implemented-by relations hip. To avoid implementation level conflation while keeping benefits of metacir \Ve present that delegation works for reifying the implemented-by rela cularity, we have developed a improved metacircular architecture, named the tionship for the CLOS MOP. In a meta-helical version of the CLOS MOP, meta helix , for the OpenC++ MOP. It addresses the problem by providing a class metaobject is responsible for defining the class that will implement two distinct ha ndles to the implemented and implementing functionality it. The class metaobject for point will produce a class point• equivalent while keeping benefits of metacircularity by using the same abstraction for to the definition: the two handles. Programmers can enjoy mctacircularity and, ncedcd if 1 (defclass point• () sw itch the handles to distinguish the implementing level from the imple (variable x y history)) mented level. The meta helix is not only for the OpenC++ MOP but also 64 CHAPTER 5. META HELIX other kinds of MOPs. To show this, this chapter also presented a meta helical version of the CLOS MOP.

Chapter 6

Libraries In OpenC++

Because of the ability for context-sensitive and non-local processing, the OpenC++ MOP makes it possible to include useful control/data abstrac tions in a library and, if it is already possible in regular C++, to implement the abstractions more efficiently. This chapter illustrates examples of li braries that the OpenC++ MOP ena bles but regul ar C++ does not. The first two examples show abstractions that the Open C++ MOP en ables, and the next example presents that we can write a class li brary for metaclasses and make it easier to write similar metaclasses. Then, we men tion that t he Open C++ MOP is also effective for meta-level programming. A few abstractions provided by the OpenC++ MOP for meta-level pro grammers are implemented by the MOP itself. Finally, we s how examples of abstractions that t he Open C++ MOP can make effi cient.

6.1 Named Object Library

The OpenC++ MOP makes it possible to implement the named object library that we presented in Chapter 2. With this libra ry, the users can get t he class name of a n object at runtime. For example, the users may write something like t his: class Complex : public NamedObject { public: double r, i; };

void f(Complex• x) { cout << 11 X is "<< x->ClassNarne() ; }

If invoked , a fun ction f() displ ays "x is Complex" . To implement this library, the developer needs to write two kinds of programs: a base-level program and a meta-level program . The base-level

65 66 CHAPTER 6. LIBRARIES IN OPENC++ 6.2. DISTRIBUTED OBJECT LIBRARY 67

program defines a library class NamedObject and it is linked with the user embedded in the constructed Ptree metaobject. Name() returns the name program . The meta-level program defines a metaclass for processing the of the class metaobject. The difference between qMake () and Make() is user program. We first show the base-level program: analogous to the difference between the back-quote notation a nd the quote notation in Lisp (As for the back-quote notation, see Appendix A). We show metaclass NamedObject : NamedObjectClass; class NamedObject {} ; the implementation of qMake() later in this chapter since its implementa tion needs the Open C++ MOP; The member function qMake () cannot be The base-level program defines a li brary class NamedObject. Also, the pro implemented in regular C++. gram declares that the metaclass of NamedObject is NamedObjectClass. To use this library, programmers first need to compile the meta-level Note that the class NamedObject does not include the member function program by the Open C++ compiler, and then they have to link the compiled ClassName(). It is automaticall y inserted to NamedObject and its subclasses code wit h the original OpenC++ compiler. Suppose that the file name of by the meta-level program. For example, the subclass Complex defined by the meta-level program is nameclass. cc: the library user is translated by the meta-level program into: class Complex : public NamedObject { I. occ -- -o myocc opencxx .a narneclass.cc public: double r, i; virtual char• ClassName() { return "Complex"; } The resu lting executable module myocc is an extended OpenC++ compiler }; with which programmers can use the metaclass NamedObjectClass. The linked archive opencxx . a is a n archive including the original Open C++ The meta-level program defin es the metaclass NamedObjectClass, wh ich compiler occ. performs t he translation mentioned above. Since a subclass of NamedObj ect The libra ry users compil e their program by myocc. For example, to inherits the metaclass from NamedObject, the metaclass NamedObjectClass com pile a source fil e complex. cc, the users may say something like this: controls the translation on the cl ass Complex as we ll although there is no ex plicit metaclass declaration. The definition of NamedObjectClass is as 'l. myocc complex.cc follows: class NamedObjectClass : public Class { public: This com mand compiles complex.cc by the extended OpenC++ co mpiler NamedObjectClass(Ptree• d, Ptree• m) Class(d, m) {} a nd produces a n executable module. Ptree• TranslateBody(Environment• env, Ptree• body){ Ptree• mf = Ptree::qMake( 11 public: 11 "virtual char• ClassName() {" 6.2 Distributed Object Library 11 11 '' return \'''Name()'\ ; n}\nu); Distributed objects are a nother example of data abstractions t hat reg ul ar return Ptree::Append(body, mf); C++ cannot include in a libritry. This section illustrates how the Open C++ } }; MOP works for including this abstraction in a library. This example needs more member fun ction s to be inserted in library users' classes by a metaclass. This metaclass overrides the member function TranslateBody() , whi ch is Hence, through this example, we illustrate how to usc various metaobjects invoked to translate members included in a class declaration. The overrid like a Typeinfo metaobject for meta-level progra mmin g. den member function just co nstructs a Ptree metaobject that represents Developing a library with the OpenC++ MOP follows three steps: (1) the member function ClassName(), and appends it to the other members. determine \\"hat a user prog ram s hould look like, (2) figure out what the user Ptree: :qMake() is a member function pro vided by the OpenC++ MOP. program should be translated into and what runtime library is needed to It constructs a Ptree metaobject accord ing to the given form at. Unlike run the translated prog ram, an d (3) write a meta-level program to perform a simil ar fun ction Ptree: :Make() , it com putes an expression a ppearin g in the translation a nd also write the necessary run tim e library. We present the the form at if the expression is s urrounded by back-q uotes ( ' ). In the exam implementation of the distributed object library in the ord er of the three ple a bove, Name() is co mputed at compile time a nd the resu ltin g value is steps. 68 CHAPTER 6. LIBRARIES IN OPENC++ 6.2. DISTRIBUTED OBJECT LIBRARY 69

What the user program should look like The client program first calls a li brary function StartupClient (), which connects the client to the server machine specified by the argum ent. In the The distributed object library helps the users write a program with dis example above, the client con nects to the server machin e named "calvin". tributed objects. The users should be able to define a distributed object as Then the client program im ports a distributed object from the server by a easy as they define a non-distributed object. library (macro) fun ction Import(). Once the distributed object is imported, For example, the users write something like this code: the client program can deal with the object in the same way that it deals metaclass Rectangle : DistributionClass; with ordinary objects. class Rectangle { public: Rectangle(int l, int h) { length = l; height = h; } What the user program should be translated into int Stretch(int l, int h) { length += l; height += h; return length * height; To run the user program shown above, the meta-level program of the li } brary needs to translate the user program and in serts marshalling and un marshalling code. The marshallin g code converts function arguments into int length, height; a byte stream so t hat lower-level function s can handle and send them to a }; remote machine. The unmarshalling code performs the reverse conversion . First, the member fun ction call appearing in the client program s hould Note that a class Rectangle turns to a class for distributed objects by just be translated so that marshalling code is inserted . The underlined code in putting a metaclass declaration. the origin al client program: Once putting the declaration, the user can create a Rectangle object on a server machine and access it from a client machine without concern of the I I client side location of the object: main() { 11 I I server side StartupClient(••calvin ); main() Rectangle* obj = Import("rect", Rectangle); { cout <<"new size: '' << obj->Stretch(1, 3); Rectangle* r =new Rectangle(3, 4); } Export(r, ''rect", Rectangle); Server Loop() ; } should be translated in to this:

This program first creates a Rectangle object and then exports the object I I client sid e main() for clients. The (macro) function Export() is a library function for exporting { a distributed object with a global name. The first argument is the exported int i; 11 object, the second argument is the global na me, and the third argument is StartupClient("calvin ); the type name of the exported object. In the example above, a Rectangle Rectangle* obj = Import("rect", Rectangle); object r is exported with a global name "rect". The function ServerLoop () cout << "new size: '' << (i = O,•(int•)&m8uf[i]=1,i+=sizeof(int), is another library fun ction , which starts waiting for requ ests from a cli ent. •(int•)&m8uf[i]=3,i+=sizeof(int), The server program has to call this libra ry func tion after all distributed CallRemote(i,obj,2)); objects a re ready. } After ServerLoop() is called, the client program can freely access the distributed object on the server machine: The substitu ted code show n by the und erlin e copies two integer a rguments I I client sid e 1 a nd 3 into an a rray of cha racters mBuf. This co pyin g involves type conver main() sion.' The variable i means that the size of the co pied a rguments. T hen, { a library fun ction CallRemote() is called for send in g the a rguments stored StartupClient("calvin"); 1 Rectangle• obj = Import("rect", Rectangle); Thc type conversion shown abo\'c assum es that all machines arc based on the same cout « "new size: " « obj->Stretch(1, 3); a•-chitccturc. Par real systcl!l s, it. shou ld absorb difference between architectures, such as } little cndian and big cndian. 70 CHAPTER 6. LIBRARIES IN OPENC++ 6.2. DISTRIBUTED OBJECT LIBRARY 71

in mBuf to the server machine. CallRemote() deals with the arguments in Write a runtime library mBuf just as a simple byte stream. Since the implementation of the mar After determining what a user program should be translated into, the library shalling code depends on the signature of Stretch(), it should be inserted developer writes a runtime library that the translated user program uses to by the meta-level program . run. It is an ordinary li brary written in regular C++ and includes such At the server side, unmarshalling code needs to be inserted. The user functions as Export(), Import(), ServerLoop(), CallRemote(), and so program should be translated into the following program: on. We show the implementation of the runtime library in Appendix C.l. The readers who are interested in details may see it.

class Rectangle { Write a meta-level program public: Rectangle(int 1, int h) { length = 1; height = h; } The rest of the work that the developer has to do is to write a meta-level int Stretch(int 1, int h) { program for the translation mentioned above. The meta-level program de length += 1; height += h; return length • height; } fines the metaclass DistributionClass, which overrides member functions TranslateSelf () and TranslateMemberCall () inherited from the default metaclass Class. TranslateSelf () controls the translation of a class def int length, height; void Dispatch(int•, void•, int); inition and TranslateMemberCall () controls the translation of a member } ; function call expression . Since the whole meta-level program is about a hun dred lines, we show it in Appendix C.l and here present just hi ghlights of void Rectangle::Dispatch(int• buf, void* obj, int member) the program. { switch(member){ The role of the overridden TranslateSelf () is to produce a member function Dispatch() and in serts it into the user program. This is the im plementation of TranslateSelf (): case 2 : I I if Stretch() is called { Ptree• DistributionClass: :TranslateSelf(Environment• env) int s 0; { int pl = •(int•)&buf[s]; Ptree• name; s += sizeof(int); Typeinfo t; int p2 = •(int•)&buf[s]; int i; s += sizeof(int); Ptree• code = nil; •(int•)buffer = ((Rectangle•)obj)->Stretch(pl , p2); for(i = 0; (name= NthMemberName(i)) !=nil; ++i){ } Ptreeid whatis = name->Whatis(); break; if(whatis != LeafClassNameid && whatis != PtreeDestructorid }; && LookupMemberType(env, name, t) } && t .Whatis() == FunctionType) { code= AppendDecoder(code, name, i, t); } } After the translation, a member fun ction Dispatch() is appended to the class Rectangle. Dispatch() is used to in vo ke a member fun ction for a AppendAfterToplevel(Ptree: :qMake( distributed object when a client program calls the member function. It 11 void 'Name()': :Dispatch(int* buf, void• object,'' receives a byte stream {buf), a pointer to the object {obj), a nd a n integer "int rnember){\n 11 indicating the called member function (member) . It unmarshals the byte "switch(member){\n 'code' }}")); strea m to function arguments according to the value of member, and in vokes return Class: :TranslateSelf(env); t he called member function with the unm a rshalled arguments. Note that } the unma rshalling code also depends on the signatu re of the call ed function a nd hence it needs to be produced by the meta-level program. T hi s mem ber function calls AppendDecoder() for every member function 72 CHAPTER 6. LIBRARIES IN OPENC++ 6.2. DISTRIBUTED OBJECT LIBRARY 73

that the translated class has. In t he while loop, each member is retri eved in to something li ke t his expression: by NthMemberN ame() and t he type of the member is examined to determine whether the member is a membe r function or a data member. The member int i; type is represented by a Type I nfo metaobject t returned by LookupMember Type(). If t he member is neit her a data member, a constructor, or a destruc (i = O,•(int•) &mBuf[i]=!,i+=sizeof(int ) , tor, t hen the member fun ction AppendDecoder() is call ed for the member. •Cint•) &mBuf[i]=3,i+=sizeof(int ) , Ca llRemote(i, obj,2)) AppendDecoder () produces a ca s e block and append s it to code. For ex a mple, AppendDecoder() produces t he followin g code for t he Stretch() me mber function: The implementation of TranslateMember Call() is similar to t he implemen tation of AppendDecoder(). It uses Type I nf o metaobjects and produces an case 2 : { expression t hat converts function argu ments to a network message stored in mBuf : int s = 0; int pl = •Cint•)&buf[s]; s += s izeof(int); int p2 = •(int•)&buf[s]; Ptree• DistributionClass::TranslateMemberCall( s += sizeof(int); Environment* env, Ptree• object, •(int•)buf = ((Rectangle•)obj)->Stretch(p!, p2); Ptree• op, Ptree• member, Ptree• arglist) } { break ; Typeinfo ftype, atype; int id = IsMember(member);

T he code produced by AppendDecoder() is included by the implementation Ptreeiter next(Ptree: :Second(arglist)); of Dispatch() , whi ch is event ually inserted by AppendAfterToplevel () Ptree+ code = nil; just after t he t ranslated class definition. Ptree• tmp = Ptree: :GenS ym(); Typeinfo metaobjects are also used in AppendDecoder() . For example, env-> InsertDeclaration(Ptree: : qMake(" int ' tmp' ; ")); t he foll ow ing for loop is part of AppendDecoder(): LookupMemberType (env, member , ftype); for(int i = 0; ftype . NthArgument(i, atype); ++i){ Type info atype; Ptree• p = next(); f or ( i = 0; t . NthArgument(i, atype); ++i ) { Ptree• tname = atype .MakePtree(); Ptree• argtype = atype.MakePtree(); code = Ptree: :Snoc(code, Ptree: :qMake ( code = Ptree: :Snoc(code, Ptree: :qMake( "* ( ' tname' * )&-mBuf [' tmp '] rr "'argt ype' p'i' = •C'argtype'•)&buf [s] ; \ n" '';;;: 'Transl ateExpression(env, p)' , 11 "s += sizeof('argtype');\n")); 11 'tmp' += sizeof('tname') , " )); } next(); I I skip , } This for loop uses Type Info metaobjects to produce the code for retrievin g 11 return Ptree: : qMake ( ( 'tmp' =0, 'code' rr argume nts from a network message stored in buf. The va ri able t is the "CallRemote('tmp', 'objec t', 'id'))'1 ); Type Info metaobject for the type of the processed mem ber function . Note } t hat the type of each argume nt is obtain ed by calling NthArgument () for t his metaobject. Mak ePtree () is a nother important member fun ction of Ty peinfo. It conve rts t he Typeinfo metaobject to a Ptree metaobject This member fun ction first inserts a varia ble declaration in t he processed t hat represents t he ty pe na me. In t he code show n a bove, MakePtree() is program by callin g Ins ertDeclaration() for env. T hi s declares a tem po used to obtain the type na me of each a rgum ent. ra ry vari able used in t he t ra nslated expression. The name of t he temporary T he meta class DistributionClass also ove rrides TranslateMemberCall (), varia ble is give n by callin g Ptree: : GenSym() . Then TranslateMemb erCa ll ( ) whi ch t ranslates a. member function call expression. For exa mple, it t rans looks up t he type of t he call ed member fun cti on and produces t he code fo r lates an ex press ion in the use r program: converting fun ction argum ents to a network message. The prod uced code is fin a ll y connected with ot her code a nd returned as the res ult of the tra nsla obj->Stretch(!, 3) tion. 74 CHAPTER 6. LIBRARIES IN OPENC++ 6.3. WRAPPER LIBRARY 75

6.3 Wrapper Library class Point { public: Since a sim il ar kind of abstraction often requires simil ar code translation, void Move (int, int); void rMove(int, int); programmers may write a library to help meta-level programming for the int x, y; similar code translation. Such a library should be called a metaclass library. public: In this section, we present an example of metaclass libraries. void wrapper_Move(int, int); A wrapper function is a useful technique for implementing abstractions void wrapper_rMove(int, int); }; such as concurrent objects. It is a function that wraps another function in itself and, if in voked, simply call s the wrapped function . But it may also void Point: :Move(int new_x, int new_y) { ... } performs some computation before or after calling it . For example: I I inserted wrapper function for Move() void Point: :wrapper_Move(int p1, int p2) int f(int i) { return i + 1; } { cout « "Move() is called. \n"; int wrap_f(int i) { Move(p1, p2); cout « "f() is called. \n" } return f(i); } void Point::rMove(int diff_x, int diff_y) { .. . }

Here, wrap..f() is a wrapper function fo r f(). It prints a message and then I I inserted wrapper function for rMove() call s the wrapped function f (). void Point: :wrapper_rMove(int p1, int p2) { A number of abstractions can be implemented by metaclasses that pro cout « "rMove () is called. \n"; duce wrappers for all member fun ctions of a class. Suppose that a metaclass rMove(p1, p2); MyWrapperClass does such a thing. This metaclass translates the user pro } gram shown below: void f(Point& p) { metaclass Point MyWrapperClass; p. wrapper _Move (3, 5); I I call the wrapper class Point { p.wrapper_rMove(-1, 2); II call the wrapper public: } void Move(int, int); void rMove(int, int); int x, y; Note that a ll occurrences of member call s for Point objects are substituted }; by the call of the wrapper fun ction. For example, the call of Move() in f () is s ubstituted by the call of the wrapper function. void Point: :Move(int new_x, int new_y) { x = new_x; y = new_y; This kind of wrapper metaclass is found in implementations of many } abstractions. The translation by those wrapper metaclasses are quite simi la r and the only difference is what the produced wrapper fun ctions perform void Point::Move(int diff_x, int diff_y) { before or after calling the wrapped fun ctions. For example, t he wrappers x +; diff_x; y +; diff_y; } s ho wn above just print a message, bu t, if they in stead perform synchro ni zation before calling the wrapped func tion , then Point objects will be void f(Point& p) concurrent objects. { p.Move(3, 5); I I call Move() p.rMove(-1, 2); I I call rMove() A meta-level program using a metaclass library } Since \\'ra pper metaclasses like MyWrapperClass arc quite sim il ar to each in to the foll ow ing program including wrapper functions for Move() and other, we s hould write a base class of these metaclasses a nd provide it as a rMove(): metaclass libra ry. Library developers can make their wrapper cl asses inh erit 76 CHAPTER 6. LIBRARIES IN OPENC++ 6.3. IVRAPPER LIBRARY 77

from the base class and focus on what the wrapper functions perform before The underlined code is the in serted declaration . To do this translation, or after callin g the wrapped function. TranslateBody () examin es each member of the class and if the member Let the name of the base class be WrapperClass. With the metaclass is a fun ction , it in serts t he declaration of the wrapper f~nction for that library, MyWrapperClass should be defined by the following simple code: member:

class MyWrapperClass : public WrapperClass { Ptree• WrapperClass: :TranslateBody(Environment• env, Ptree• body) public: { MyWrapperClass(Ptree• d, Ptree• m) : WrapperClass(d, rn){} Ptree• decl = Ptree : :qMak e("public: \n") ; Ptree• WrapperBody(Environment•, Ptree•, Ptree•, int, Ptree* name; Typeinfo&); Typeinfo t; }; int i = O· while((n~e = NthMemberName(i++)) != nil){ Ptree• WrapperBody(Environment• e, Ptree• name, Ptree• wrapper, Ptreeid whatis = name->Whatis(); int nargs, Typeinfo& ftype) if(whatis != LeafClassNameid { && whatis != PtreeDestructorid Ptree• body= Class: :WrapperBody(e, name, wrapper, nargs, && LookupMemberType(env, name, t) ftype); && t.Whatis() == FunctionType){ return Ptree::qMake( Ptree• m = t.MakePtree(WrapperName(name)); 11 11 11 Cout << \ 'name'() is called.\nV•;'body'••); decl = Ptree: : qMake ( 'decl' 'm' ; \n 11 ) ; } } }

Note that the metaclass MyWrapperClass inherits from WrapperClass and return Ptree: :Append(body, decl); ove rrides only a member function WrapperBody(), which produces the body } of a wrapper function. The base cl ass performs the rest of the translation, which is to insert the declarations of wrapper fun ctions, to replace all oc Here, the member fu nction WrapperName() a member function of the class currences of member calls with calls of the wrapper functions, and so forth. WrapperClass. It returns the name of the wrapper function for the given member fun ction. A metaclass library The other member fun ction overridden by WrapperClass is Translate MernberFunctionBody() , which translates the body of a member function. T he metaclass WrapperClass provided by t he metacl ass library needs to It is ove rridden to produce the definitions of wrapper functions. For exam execute three things: (1) to insert wra pper fun ctions in the user program, (2) ple, TranslateMemberFunctionBody() processes t he definition of a mem to substitute call s of wrapper fun ctions for calls of the wrapped functions, ber function Move() and inserts the definition of the wrapper fun ction a nd (3) to provide a member fun ction that a subclass of WrapperClass wrapper _Move() ri ght after Move(): can override for specifying the behav ior of wrapper functions. We below present overviews of how the three things are implemented. The complete void Point: :Move(int new _x, int new_y) { implementation of WrapperClass is found in Appendix C.2. x = new_x; y = new_y; For (1), WrapperClass overrides two member fun ctions. One is Trans } lateBody() , which controls the translation of a class definition. It is over ridden to in sert the declarations of wrapper fun ctions in the cl ass definition. void Point: :w rapper_Move(int pl, int p2) { For example, the defi nition of t he cl ass Point is t ra nslated into this code: cout « "Move() is called. \n"; Move(pl, p2); class Point { } public: void Move (int, int) ; int x, y; TranslateMernberFunctionBody() constructs t he defi ni tion of wrapper public: _Move() frolll the Typeinfo lll etaobject for the type of Move(). l'irst , it ~ id wrapper_Move(int, int); derives a rgn lll ent types and the return type from that Typ e info llletaol> }; ject and converts those types to Ptree metaobjec ts by callin g MakePtree(). 78 CHAPTER 6. LIBRARIES IN OPENC++ 6.4. IMPLEMENTATION OF QMAKE() 79

Then it assembles the converted Ptree metaobjects with the wrapper name 6.4 Implementation of qMake() and the body of the wrapper function, and constructs the complete definition of wrapper ...Move (). The overall structure of TranslateMemberFunct ionBody () The OpenC++ MOP provides a member function Ptree: :qMake() (quoted is as follows: make), whi ch constructs a Ptree metaobject according to the given format. Although this member function is more convenient than a simi lar mem Ptree• WrapperClass::TranslateMemberFunctionBody( . . . ) ber fun ction Make() , it cannot be implemented within regu lar C++, but { requ ires meta-level programming to be implemented. Without meta-level programming, only Make() is available. Ptree• arglist = argu ment list of the wrapper function The implementation of qMake () is an example of meta-meta level pro gramming in OpenC++. T he OpenC++ MOP uses itself to implement Ptree• body WrapperBody(env, name, wrapper_name, i- 1, t); metaobjects and their member functions such as qMake (),so that t he Open C++ Ptree• head Ptree: : qMake ( MOP provid es better abstractions and programming in terface fo r program '''Name()' : : 'wrapper_name ' ( 'arglist ' ''); mers to write metaclasses. T hrough this example, we present that Open C++ can naturally deal with meta-meta level programming a nd it actually uses AppendAfterToplevel(Ptree :: Make(" 'head'{'body'}\n")); meta-meta level programming for implementing abstractions that make it return Class: :TranslateMemberFunctionBody( .. . ); easier to write metaclasses. }

What the user program should look like The produced definition of wrapper ...Move() is inserted by calling Append AfterToplevel (). Note that Trans lateMemb erFunctionBody() calls Wr ap Recall the usage of qMake (). If a variables tmp is a pointer to a Ptree perBody() to make a fun ction body so that (3) a subclass of WrapperClass metaobject representing a variable name xyz, an d a vari able i is a n in teger can override it and specify the behavior of wrapper functions. The defaul t 3, then programmers may write something like this: WrapperBody() s upplied by WrapperClass returns an expression that just 11 calls the wrapped function. Ptree* exp = Ptree: : qMake ( int 'tmp' = 'i' ; 11 ) ; Finally, we show TranslateMemberCall () , whi ch WrapperClass over rid es for (2). It substitutes call s of wrapper fun ctions for calls of t he wrapped This program constructs a Ptree metaobject "int xyz = 3" . The expres functions. To do this substitution , it just call s TranslateMemberCall() sions su rrounded by back-quotes are expanded when qMake() is in vo ked. s upplied by Class with the name of a wrapper function: The program shown above can be rewritten in to a progra m usin g Make ():

Ptree• exp = Ptree: :Make("int 'l.p = 'l.d", tmp, i); Ptree• WrapperClass::TranslateMemberCall(Environment• env, Ptree• member , Ptree• arglist) { Unlike qMake() , Mak e() takes a format and some parameters, which are return Cl ass::TranslateMemberCall(env, WrapperName (member), substituted for the occurrences of 'l.p a nd 'l.d in the form at. This kind of arglist); programming interface is popular in C and C++, but it becomes error } prone as the number of parameters increases. Typical errors caused by this interface are to give a wrong number of para meters and to place parameters Note that the second argument to TranslateMemberCall () supplied by in a wrong order. Class is no t the name of the wrapped fun ction, that is, member, but the name of the wrapper function. Thus a member function call for a wra pper class object: What the u ser progra m s hould b e translate d into Function ca ll s of qMake() a re translated in to a combin ation of several func p.Move(3, 5) tion ca ll s. For example, this program : is translated into this ex pression: Ptree• exp = Ptree: :qMake("int 'tmp' Ci C; II); p.wrapper_Mov e(3, 5) is translated in to: 80 CHAPTER 6. LIB RARIES IN OPENC++ 6.5. METACLASS 81

11 11 11 11 11 Ptree• e xp ;:; (Ptr ee* ) (Pt r eeHe ad () + int +tmp+ = ''+i+ ; ) ; els e r etur n Ptree: :Fi rst(Ptree: :Last(qual ified_name)); Note t hat t he program after t he t ranslatio n is in regular C++. For ex } ample, t he character st ring "tmp" becomes a vari able name tmp after the translation . Since regular C++ cannot convert a cha racter string to a va ri After t hat, Transl ateMemberCall () determi nes whether the member name able name at ru ntime, the con version shoul d be done by the translation at is qMak e, and if so, it converts t he first argument from a Pt ree metaobject compile time. to a cha racter stri ng. This conversion is done by calling Re ify() for ar g l. Ptr eeHead() returns a PtreeHead object, wh ich is a stream object pro Then TranslateMemberCall() call s ProcessBackQuot e() wit h the con ducing a Ptree metaobject. T he + operator is used to input a ch aracter ve rted character stri ng to t ra nslate the member-call expression. Pr ocess string, a Ptree metaobject , and so o n, to the stream object. The inputted BackQuot e() is a me mber function of QuoteClass. data are concatenated into a Ptree metaobject, and the concatenated Ptr ee metaobject is obtain ed by explicit ly casting t he PtreeHead object into t he 6.5 Metaclass type Ptree•. T he cast operator is ove rloaded to return t he concatenated Ptree metaobject. Only the implementation of qMake () is not an example of use of meta-meta level programming. T he defa ul t metaclass Class and its s ubclasses a re also Write a meta-level program implemented with using meta-meta level prog rammi ng , so t hat progra mmers can easil y defin e a new metaclass. For this reason , the metaclass Class Wri ting a metaclass fo r performing the translation ment ioned above is quite a nd its subclasses a re also class metaobjects, and they a re insta nces of the straightforward. To do this t ranslation, the metaclass QuoteClass just over metacl ass Metaclas s . F igure 6.1 shows t his instance-of relationshi p. rides a member fun ction TranslateMemberCall ():

Ptree• QuoteClass: :TranslateMemberCall(Environment• env, subcl ass- of Ptree• member, Ptree+ args) { ·~ ~ Ptree• name SimpleName(member); Metaclass ...... Class ...... Point ...,. .... · pO char* str; in stance- of if (Ptree : : Eq (name, "qMake")) { Ptree• arg1 = Ptree: :First(Ptree : :Second(args)); Fig ure 6.1: T he instance-of relationship amo ng metacl asses if(arg1->Reify(str) && str != nil) return Pr ocessBackQuote(FALSE, str); el se ErrorMessage("bad argument for qMake()" , arg1); } P r otocol w ithout meta-meta level p r ogramming else return Class: :TranslateMemberCall (env, member, args); T he defini t ions of metacl asses seen so far have not expli cit ly showed a ll t he } protoco l that the metacl asses must obey. They need to be translated by t he metaclass Metaclas s so t hat t hey satisfy a ll the protocol. This member fun ction translates a given fun ction-call expression if the called To obey a ll t he protocol, (i) a new metacl ass has to have a member fun ctio n is qMake(), otherwise it delegates the tra nslation to the member fun ction MetaclassName O, a nd (ii ) a fun ction that instantiates t he meta fun ction supplied by the base class Class . cl ass mu st be registered. Fo r example, a new metacl ass MyC l ass should be Sin ce the argum ent member may be not a simple member na me bu t a translated in to somethin g more complex tha n what we have seen: q ualified na me such as Ptree : : qM ake( ) , TranslateMemberCall 0 first call s SimpleName () to strip t he class na me and do uble colons o iT: cla ss My Cl ass publ i c Cl ass { public: Ptr ee• Qu oteClass: :SimpleName(Ptree • qualified_name) MyClass (Ptree• d, Ptree• m) : Class (d, m) {} { if(qualified_name->I sLeaf()) r eturn qualified_name; char~ Met a className () { r eturn "MyClass" ; } 82 CIIAPTER 6. LIBRA lUES IN OPENC++ 6.6. VECTOR LIBRARY 83

} ; AppendAfterToplevel(Ptree::Make( "static Class• 'l.p(Ptree• def, Ptree• marg){\n" static Class• CreateMyClass(Ptree• d, Ptree• m) " return new 'l,p(def, marg); }\n" { "static ListDfMetaclass 'l.p(\"'l.p\", 'l,p,\n" return new MyClass(d, m); " 'l.p:: Initialize()); \n", } tmpname, name, Ptree: :GenSym(), name, tmpname, name)); static ListOfMetaclass myClassDbject("MyClass", CreateMyClass, MyClass::Initialize()); return Class::TranslateSelf(env); } After the translation, a member function MetaclassName(), a function Ptree• Metaclass: :TranslateBody(Environment• env, Ptree• body) CreateMyClass(), and an object myClassObject are in serted. CreateMy { Class () is a function that instantiates the metaclass, and the object my Ptree• mem = Ptree: :Make("public: char• MetaclassName() {\n" 11 11 ClassObj ect is created at the beginning of runtime and registers CreateMy •• return \"'l.p\ ; } , Class (). The registered fun ction is used to implement a function that Name()); return Class: :TranslateBody(env, Ptree: :Append(body, mem)); receives a class name in the form of character string and instantiates the } specifi ed class. This function is internally used by the OpenC++ compiler when in stantiating a metaclass, because the new operator does not take a The member function TranslateSelf() inserts a function and a variable for character string to specify the instantiated class: instantiating the metaclass, and TranslateBody() inserts a member func tion MetaclassName () in the declaration of the metaclass. For the reason of class Point { }; .. . bootstrapping, Metaclass uses not Ptree: : qMake() but Ptree: :Make() . Point• pO = new Point; II valid char* c = 11 Point"; Since a su bclass inherits the metaclass from the base class, programmers Point• pl = new c· II invalid do not need to expli citly write a metaclass declaration for new metaclasses. The OpenC++ compiler automaticall y selects Metaclass for the new meta Note that new Point is a valid expression but new c is not since c is not a classes since they are subclasses of Class. No ad-hoc implementation is re class name but a variable. quired; the OpenC++ MOP can naturally deal with this and programmers The behavior implemented by the inserted functions and variable can can enjoy a simpler protocol by this mechanism. not be inherited from the base class; it must be explicitly implemented by every new metaclass. This is because implementing the behavior needs the 6.6 Vector Library definition of the new metaclass as the exam ple of named object that we have shown in Chapter 2. All the examples shown above are of abstractions that regular C++ cannot handle but the OpenC++ MOP can do. The OpenC++ MOP also makes Protocol with meta-meta level programming some kinds of abstractions more efficient than in regul ar C++. From this Sin ce the definition obeying all the protocol is complex and error prone, section, we show a few examples of s uch abstractions. the actual OpenC++ MOP has meta-level programmers wri te a simpler The first example is the vector li brary. In Chapter 2, we showed that definition of a metacl ass, and automatically translates it into the regular the template mechanism of C++ enables a vector abstraction for a ny type, definition that obeys all the protocol. To do this, the OpenC++ MOP but t he implementation with t he template mechanism was not as efficient provides a metaclass Metaclass, which is the metaclass of all metaclasses. as an id eal implementation. If the OpenC++ MOP is used, however, the vector abstraction is implemented more efficiently. Like other metaclasses, the metaclass Metaclass is a subclass of Class. It overrides TranslateSelf () and TranslateBody (): Vector library in regular C++ Ptree• Metaclass::TranslateSelf(Environment• env) { As we showed in Chapter 2, a vector abstraction is implemented in regular Ptree• name= Name(); C++ by the following template: Ptree• tmpname = Ptree: :GenSym(); template class Vector { 84 CHAPTER 6. LIBRARIES IN OPENC++ 6.6. VECTOR LIBRARY 85

T elements [SIZE] ; return TemplateClass::TranslateAssign(env, object, public: op, expr); Vector operator + (Vector& a, Vector& b) { Ptree• index; Ptree : :GenSym (); Vector c; r eturn Ptree ::qMakeStatement( for( i ; 0; i < SIZE; ++i) "for(int 'index' = 0; ' index' IsLeaf()) vector abstraction needs to be implemented wit h very generic description - r~turn Ptree: : qMake ( "' expr' . element ['index'] 11 ) ; overloadin g primitive operators like +. No implementation techniques for a else lf(Ptree: :Match(expr, "['l. ? + 'l.?]", &lexpr, &rexpr)) return Ptree: : qMake(" 'In line (env, lexpr, index)''' particular case can be included in the description. "+ ' Inline(env, rexpr, index)'''); else if(Ptree::Match(expr, "['l.?- 'l.?]", &lexpr, &rexpr)) 11 Vector library in OpenC++ return Ptree: : qMak e ( 'Inl ine ( env, lexpr, index) ''' "- 'In line (env, rexpr, index)'''); If the O pe nC++ MOP is used , the vector abstraction can be implemented else if(Ptree::Match(expr, "[( 'l,? ) ]", &lexpr)) return Ptree: : qMake (" ( 'Inline ( env, l expr, index) ') " ) ; more efficiently. The library developer can define a special metaclass for else if (Ptree: :Match(expr, " [- 'l.?]", &lexpr) ) t he template class Vector, wh ich t ra nslates successive operators in to a n return Ptree : :qMake(''-'Inline(env, lexpr, index)'''); efficient sin gle loop instead of separate fun ction calls. Doing this translation else if(Ptree: :Match(expr, "['l.? • 'l.?]" , &lexpr, &rexpr)) return Ptree: :qMake( is straightforward ; only one member function TranslateAssignO needs to 11 '"lexpr' * 'Inline(env, rexpr, index)' ); be ove rridden : else{ 11 11 ErrorMessage( invalid vector expression , expr); #include 11 template.h 11 return nil; } class VectorClass : public TemplateClass { } public: VectorClass(Ptree• d , Ptree• m) : TemplateClass(d, m) {} Ptree+ TranslateAssign(Environment+, Ptree•, Ptree+, The member function Inline 0 tests whether the given ex pression matches Ptree•); a pattern and , if it matches, calls Inline() rec ursively to process the sub Ptree• Inline(Environment•, Ptree+, Ptree+); expressions. Inline() can deal with not only the+ operator but also the }; a nd • operators and parentheses (). Ptree• VectorClass: :TranslateAssign(Environment• env, Ptree+ object, Ptree• op, Ptree• expr) Experiments { if ( !object->IsLeaf() II !op->Eq( ';') The metaclass VectorClass t ranslates an assignment expression on a vector II expr->IsLeaf()) // e .g. a; b; into an effi cient for loop. for exam ple, th is program: 86 CHAPTER 6. LIBRARIES IN OPENC++ 6.7. THE STANDARD TEMPLATE LIBRARY 87

The results of the experiment are li sted in Table 6.1. The vector library Table 6.1: Execution performance of the vector library (psec.) implemented with the Open C++ MOP achieved as good performance as the hand-coded program. Although the OpenC++ program is slightly slower length 8 length 64 than the hand-coded program when the number of vectors is 1 and the #of vectors 2 3 4 2 3 4 vector length is 64, this fact is caused by the implementation difference in Sun C++ 0.5 1.5 3.0 4.4 3.3 10.1 20.0 30.1 copying a vector. The hand-coded program copies a vector by explicitly copying each element, but the OpenC++ program copies it by using the GNU C++ 0.3 1.7 3.1 4.5 6.9 21.7 36.2 51.1 default object copy mechanism, which is compiled into a call of memcpy(). Open C++ 0.3 0.9 1.3 1.7 6.9 6.5 9.8 13.1 The implementation strategy of copying a vector is also the reason that Sun C++ achieved the best performance when the number of vectors is 1 and Hand-coded 0.9 0.9 1.3 1.7 6.5 6.5 9.8 13.0 the length of each vector is 64. The Sun C++ compiler inlines memcpy() Average of 1,000,000 (size 8} or 300,000 (size 64) iterations. when an object is copied. SPARC Station 20/61, SunOS 4.1.3 This experiment also shows that real C++ compilers do not perform the optimization that the metaclass VectorClass performs. Although an ideal compiler shou ld automatically perform the optimization, it seems difficult for real compilers to do that within reasonable space and time. Our claim Vector vi, v2, v3, v4; is that such an optimization should be done by a metaclass rather than a compiler's optimizer. Such an optimization is difficult for a general-purpose vl = v2 + v3 + v4; optimizer as our experiment showed, but on the other hand, it is not difficult for a metaclass because the metaclass is written by the library developer, is translated into this: who knows semantic information about the library code. We believe that a Vector vl, v2, v3, v4; compil er's optimizer should focu s on general optimi zations and metaclasses should perform a special optimization that is effective on ly for a particu lar class. Although forcing end-programmers to write a metaclass is not for(i = 0; i < SIZE; ++i) vl.elements[i] = v2.elements[i) + v3.elements[i] reali stic, we beli eve that it is acceptable that library programmers write + v4.element(i]; metaclasses since they are usually experienced programmers and their code is reused by many end programmers. This translation drastically improves execution performance. To show the improvement, we ran a micro benchmark program a nd measured execu 6. 7 The Standard Template Library tion time of Vector expressions. The micro benchmark program computes the sum of various numbers of vectors of double. First, we ran the program The Standard Template Library (STL) [44) is another example of abstrac with the vector li brary written in regular C++, then ran the same program tions that The OpenC++ MOP can implement more efflciently. STL is a with the vector li brary written in OpenC++. Since different compilers per C++ li brary included by the ANS I standard of C++. Despite of the high form different optimization techniques, we used GNU C++ 2.7.2.1 (with level abstractions that STL provides, a program using STL is often slower option -03) and Sun C++ 3.0.1 (with option -fast) for regular C++. The than an eq ui valent program written without STL. The OpenC++ MOP OpenC++ compil er uses GNU C++ for the backend co mpiler. Moreover, con tribu tes to avoid performance drawbacks caused by STL with keeping we ran a program that is eq ui valent to the micro benchmark but optimized its hi gh-level abstractions. by hand without the vector library. This hand-coded program uses a rrays of double instead of objects and was compiled by GNU C++. When mea Brief Ove rview of STL su ring the execution time, we changed the length of each vector between 8 and 64, and also the number of the vectors in the assigned expression from ;\ unique feature of STL is that the library consists of independent com 1 to 4. \Vh cn the number of the vectors is 1, the expression is vO = vl; 1\o ponents and the users can flexibly combine the components to obtain the addition is executed. All the benchmark programs used in this experiment functionality they need. T he main components of STL a rc containers and are in Appendix C.3. generic algorithms. The contain ers arc objects that store a collection of 88 CHAPTER 6. LIBRARIES IN OPENC++ 6. 7. THE STANDARD TEMPLATE LIBRARY 89 other objects, and the generic algorithms are functions that process con { tainers. Since type names are parameterized with using t he template mech int n = 0; while(first 1 = last) anism, STL use rs may use a generi c algorithm with a ny kind of containers. if(•first++ == value) They do not have to use a different version of the generic algorithm for a ++n; different kind of containers. For example, STL users may write something like this: return n · } list a1 = ... ; set a2 = ... ; The template argument I is the type of iterators and Tis the type of con n1 = count(a1.begin(), a1.end(), 3.14); tainer elements. Note that the variables first and last are used as if they n2 = count(a2.begin(), a2.end(), 3.14); are pointers to an array of the type T. This program computes the number of 3.14 stored in containers a1 and a2, respectively. a1 is a list container a nd a2 is a set container. count is a Performance improvement by the OpenC++ MOP generic algorithm to determine the number of elements in a contain er that Although iterators give great fl exibility to STL, they also involve perfor are eq ual to a given value. It takes pointers to the first and the last element mance drawbacks if co mpared with an equ ivalent program written without in the container and the valu e that it counts the number of. Note that the iterators. Since a generic algorithm must indirectly accesses elements in a same generic algorithm count is used for two different kinds of containers container through an iterator, its execution performance tends to be slower. list and set. The single generic algorithm serves all kinds of containers. If programmers give up generality of the generic algorithm and speciali ze The connectivity between containers and generic algorithms is enabled the algorithm to work only for a particular kind of container, then the spe by another kind of STL component called iterators. In the progra m a bove, cial ized algorit hm will be more efficient because iterators are not needed any iterators are the values returned by a1.begin0 and a1.end(). They a re more. For example, the followin g function is a speciali zed count algorithm pointer-like components that all kinds of containers provide as common for counting elements only in a list container: interface to access the elements. Generic algorithms use the iterators to traverse elements stored in a container. For example, iterators for list int count(List• first, List• last, int value) contain ers a re defined as follows: { int n = 0 · class iterator { while(fir~t != last){ public: if(first->val ue value) iterator(list• p) { ptr = p; } ++n; list* ptr; int eof() { return ptr == 0; } first = first->next; int operator 1 = (iterator& a) { return ptr != a.ptr; } } T operator * () { return ptr->value; } . iterator& operator ++ (){ptr = ptr->next; return •thls;} return n· iterator operator ++ (int) { } iterator prev = •this; ptr = ptr->next; return prev; Note that now the variables first and last are not iterators but actual } pointers to list objects. Hen ce reading an element is done by the-> }; operator instead of the * operator. Also, the ++ operator is replaced with the expression first = first->next. lterators a re objects for which pointer operators such as * a nd ++ are over The OpenC++ MOP redu ces the overheads by iterators. In OpenC++, loaded. Generic a lgorithms usc the iterators as if t hey are C++ pointers to the STL implementor can write a metaclass that specia li zes a generic a l a rrays; for example, the next template function is an implementation of the gorithm for a particu lar kind of container a nd translates a program to use generic count algorith m: that specialized algorithm. Suppose that a program uses t he generic count template algorithm with a list container. The metaclass co nverts iterators int count(I first, I last, T value) for list into actual pointers to list a nd it replaces calls of 90 CHAPTER 6. LIBRARIES IN OPENC++ 6.8. THE OOPACK BENCHMARK TEST 91

the generic count algorit hm with calls of a count algorithm speciali zed for Table 6.2: Execution performance of STL (m sec.) list . First, the defin ition of iterators for list co ntainers:

class iter ator { ... } ; list (ratio) s et (ratio)

is translated from a class type into this pointer type: Sun C++ 57 (1.73) 860 (1.39) GNU C++ 45 (1.36) 750 (1.21) typedef list * iterator ; Open C++ 36 (1. 09) 680 (1.10) T hen the tem plate fun ction show n below is derived as a speciali zed algo rithm Hand-coded 33 (1.00) 620 (1.00) fro m t he ge neri c count algorit hm. It is substit uted for the generi c count algorithm called with list contain ers: Average of 100 (l ist) or 10 (set) iterations. SPARC Station 20/514, Solari s 5.3 t emplate int count_for_list_int(I first, I last, T value) { list* trnp; int n = o· while(fir~t != last) MOP, the overh eads are redu ced to only 9%. As for s et containers, the if ( (trnp=first,first=first->next,trnp)->value == value) generic count algori t hm invol ves 39% (Su n C++) or 21% (G NU C++) over ++n; heads. Bu t t he OpenC++ MOP reduces the ove rh eads to 10%. These re su lts show that generic algori thms ofSTL work for a ny ki nd of contain ers bu t return n ; } t his adaptability causes se ri ous perfo rmance degradation. T he OpenC++ MOP recovers t his perform ance degradation from one half to one fourth This tem plate fun ction s upposes that I is bound to list*. while kee ping the adaptabili ty of STL. The translation mentioned above is easily implemented with about 40 lin es of meta-level progra m. The metacl ass for this tra nslation is effecti ve not o nl y for the generi c count algo ri t hm . It deals wi th combin ations of list contain ers and a ny ge neric algorit hm . For t he complete defini tion of the metaclass, see in Appendi x C.4 . We also s how a metaclass for s et 6.8 The OOPACK Benchmark Test co ntaine rs in the same place. The last exa mple is the OOPAC I< benchmark test [49) . This benchm a rk Experiment test is a progra m for testing the ability of a C++ co mpil er for compili ng a program wri tten in object-oriented programming (OOP) as effi ciently as a To illu strate the perform ance improvemen t by the metacl asses, we measured progra m in non-OOP. T he prog ram con tain s a s ui te of tests, each of whi ch t he execution tim e of t he ge neri c count algorit hm with/withou t t he meta consists of two eq ui valent ro utin es wri tten in OOP a nd non-OOP. T he OOP cl asses. As for t he contain ers, we used list a nd set. When the rout ines a re written with hi gher-level abstractions, whereas the non-OOP metaclasses were not used , t he measured progra m was compiled by G NU routin es are wri tten in C style for effi ciency. In ot her wo rd s, the non-OOP C++ 2.7.2 with option -0 3 a nd Sun C+ + 4.1 with option -fast . When the rou t in es a re hand-optimi zed versions of the corresponding OOP routines. metaclasses are used, the program was compiled by the OpenC++ com pil er, whic h uses GNU C++ for the backend co mpiler. Also, we measured If the OOP routines a re slower th a n t he non-OOP routines, t hat perfor the execution t im e of ha nd-optimized count algori t hms for list a nd mance degradation means costs to use hi gher-level abstractions implemented s et. T hese algorithms we re co mpiled by G U C++. All the programs with objects under the compil er. A C++ co mpile r should be able to co m used in t his experiment a re s how n in Appendix C.4. pile t he OOP rout in es as effi cie nt as the no n-OOP routines sin ce it can in T he resul ts of this experim ent is li sted in Ta ble 6. 2. As for list con prin ciple t ransform the OOP routi nes into the no n-OOP rout in es by inlin tainers, the ge neri c count algo ri t hm invol ves 73% (S un C++) or 36% (GNU in g member fun ctions and performin g constant pro pagation and strength C++) ove rh eads against the hand -o ptimi zed ve rsion, but wi th t he O pe n C++ reduction. 92 CHAPTER 6. LIBRA RIES IN OPENC++ 6.8. THE OOPACK BENCHMA RK TEST 93

Improvem ent with the OpenC ++ MOP Here, c. f or each(i ) means iterating the following block statement for each row of t he matri x c. i is bound to the ind ex of t he row currently processed. Real C++ compil ers have di ffic ul ty in compiling the OOP routines effi ciently In the block statement, c(k) indicates the k-th colu mn in the row. Unl ike as we show later, but this inefficiency is fairly recovered by the OpenC++ the for statement, the for each statement explicitly in dicates that the loop MOP. Although the OpenC++ MOP cannot im prove the execution speed is executed for traversing the rows of a matri x, so that t he statement can wit hout changing the benchm a rk progra m, it can extend the language sy ntax be translated in to optimized code fo r the t raversing. In fact, t he metaclass to all ow the programmer to write more effi cient OOP code tha n t he OOP for Matrix t ra nslates t he program above into t his efficient one: routines in the original benchmark program . T he extended syntax is used for ann otating com pil ation hin ts wi t hout directly describing lower-level details void MatrixBenchmark: :oop_style( ) canst of the implementation. It does not affect the level of abstraction of the OOP { routines . Matrix c(L, L, C); Matrix d(L, L, D); For example, the following code is the Matrix test in the benchmark. It Matrix e(L, L, E); computes mul tiplication of two matrices in OOP style: for ( int i=c.rows, t1=c .cols, t2=(c.rows -1) •c.cols; void MatrixBenchmark: :oop_style() const --i >= 0; { t2 -= t1) { Matrix c(L, L, C); Matrix d(L, L, D); double canst• t3 = &(c.Data())[t2]; Matrix e(L, L, E); for(int j = 0; j < e.cols; ++j){ for(int i = 0; i < e.rows; i++) double sum = 0; for(int j = 0; j < e.cols; j++){ for(int k=d.rows, t4=d.cols, t5=(d.rows-1)•d.cols; double sum= 0; --k >= 0; for(int k=O; k

IteratorResult sum; for(int k ; 0; k < N; k++) } Y[k] ; Y[k] + factor • X[k] ; } Here, A and B are arrays of double, and N is the length of the arrays. The Open C++ MOP provides a statement for each for the class Iterator. Here, X and Y are arrays of Complex objects. With the foreach statement, the program above is rewritten in to this: The execution performance of this test can be improved by a similar technique that we showed for the vector library in Section 6.6 . No extended void IteratorBenchmark: :oop_style() const syntax is needed. The program shown above is translated by a metaclass { into this: double sum ; 0; Iterator ai(A,N), bi(B,N); void ComplexBenchmark::oop_style() const ai.foreach(v){ { sum+; v • bi.look(); double factor_re ; 0.5; bi .next(); double factor_im; 0.86602540378443864676; }; for(int k ; 0; k < N; k++){ Y[k] .re Y[k].re+factor_re•X[k].re-factor_im•X[k].im; IteratorResult sum; Y[k] .im; Y[k] .im+factor_re•X[k] .im+factor_im•X[k].re; } } } The foreach statement for It erator provides a control abstraction simil ar to foreach for Matrix. It iterates the following block statement for each Note that a Complex object factor is broken down into two double vari element of the vector. In the block statement, a specified variable (v in the ables, factor..re and factor_im. This leads a C++ compiler to all ocate program above) indicates the element currently processed. However, the factor on registers rather than a stack frame, and eventually contributes to foreach statement for Iter a tor is translated differently way from Matrix. performance improvement. Without this translation, C++ compilers such This is oop...style() after the translation: as GNU C++ and Sun C++ do not allocate objects on registers even though allocating them is possible in principle. void IteratorBenchmark : :oop_style() const { double sum ; 0; Experiments Iterator ai(A,N), bi(B,N); for(int t7; 0, t8; ai.Limit(); t7 < t8; ++t7){ We measured execution time of the OOPACK benchmark test under differ const double& v; ai . Array(t7); ent settings. \Ve first ran the benchmark program compiled by Sun C++ sum+; v • bi.look(); 4.1 with -fast option, and t hen the program compiled by GNU C++ 2.7.2 bi.nextO; with -03 option. We also compiled the program by the OpenC++ compiler } ; with metaclasses for the translation shown above, and measured the exe IteratorResult sum; cution time of the compiled code. When compilin g with those metaclasses, } the benchmark program was rewritten to utilize the extended syntax. The OpenC++ compiler used GNU C++ 2.7.2 with option -03 for the back The length of the vector that ai points to is stored in a local variable before end compiler. All the programs used for the experiment are presented in starting iteration. This eliminates accesses to ai when the loop-termination Appendix C.5. condition is checked . The resu lts of the experiment are li sted in Table 6.3. The OOPACK The last test in the benchmark is Complex. This test computes a complex benchmark consists of four tests: Max , Matrix, Complex, and Iterator. We valued "SAXPY" operation, and measures how efficientl y a C++ compil er did not develop a metaclass for the Max test, which is for measuring how compiles a Complex object, which represents a complex number and is very well a C++ compiler inlines a function . !3ut for the remaining three tests, popular in scientific computing: OpenC++ showed better performance than Sun C++ and GNU C++. If OpenC++ is used, the OOP routines involves only 10% to 20% overheads void ComplexBenchmark: :oop_style() const { against the non-OOP routines written in C style. On the other hand, the Complex factor(0.5, 0.86602540378443864676); OOP routines compil ed by Sun C++ is twice or three times slower than the 96 CHAPTER 6. LIBRARIES IN OPENC++ 6.9. SUMMARY 97

class writers. Its interface cannot be implemented without the OpenC++ Table 6.3: Execution time of the OOPACK benchmark (sec.) MOP. Metaclass is a meta-metaclass for all metaclasses. It simplifi es the protocol for writing a new metaclass. Sun C++ GNU C++ Open C++ The remaining three examples showed that the Open C++ MOP can be Max (C-style) 8.1 9.0 9.0 used to improve execution performance of some kinds of abstractions. The (OOP) 8.5 12.0 12 .0 OpenC++ MOP makes it possible to specialize library code for user code Ratio 1.1 1.3 1.3 and improve execution performance. This specialization includes elimination of unnecessary indirection and encapsulation and it helps C++ compilers Matrix (C-style) 10.9 9.8 9.8 to generate more efficient object code. Furthermore, the OpenC++ MOP (OOP) 31.3 80.7 11.2 allows syntax extension, which is used for putting annotations for efficient Ratio 2.9 8.2 1.1 compilation. In the example of the OOPACK benchmark, we showed that Complex (C-style) 13 .2 11.2 11 .2 such syntax extensions actually improve execution performance while keep ing the level of abstraction . (OOP) 23.3 18.2 13 .1 Ratio 1.8 1.6 1.2 lterator (C-style) 7.1 7.1 7.1 (OOP) 15.2 8.2 8.1 Ratio 2.1 1.2 1.1 50000 (Max), 500 (Matrix) , 20000 (Complex), 50000 (lterator) iterations. SPARC Station 20/514, Solaris 5.3

non-OOP routines. The OOP routines compiled by GNU C++ also involves 60% or 20% overheads for Complex and Iterator, but the Matrix test is more than 8 times slower.

6.9 Summary

This chapter presented eight examples of libraries th at the Open C++ MOP enables. The first two examples, named objects and distributed objects, pre sented useful data abstractions that regular C++ cannot handle. Through the examples, we also illustrated how various metaobjccts like Type Info are used in meta-level programming. The next example, wrapper library, is an example of metaclass libraries. Implementing abstractions like named objects and distributed objects is facilitated if there is a metaclass library which provides typical meta code for metaclass writers. This example showed a metaclass WrapperClass, which helps programmers write a metaclass handling wrapper functions. qMake() and Metaclass are examples of meta-meta level programming. The OpenC++ ~ fOP natura ll y allows meta-meta level programming in or der to make useful abstractions available at meta level as well as base level. qMake() is a meta-level function providing convenient interface for meta-