OpenJava – a Class-based Macro System for Java

Michiaki Tatsubori, Shigeru Chiba, Marc-Olivier Killijian, Kozo Itano

OOPSLA'98 Workshop on Reflective Programming in C++ and Java Why Metaprogramming

● Programming is sometimes tedious – Repetitive tasks – Error-prone tasks

● The computer is good at this kind of work!

● Idea: The programmer writes both the program and the metaprogram – Metaprogram is the program that processes the programmer's program at compilation time – Metaprogram can use metadata to guide the processing Example 1: Safe SQL

Code Code class Employee { “select emp, manager”

m

bool filter(Query q) { a if (q.empID > r “where emp.id >

g

d.manager.emplID) o manager.id ... r AND } p ...... ta “

e };

M

Requires code analysis, complex logic and code generation.

=> Need code to generate code! Example 2: Processor Simulator

Metadata Code

Inputs Outs Impl FU m

a class ADDInstruction { ADD RS,RT RD ... ALU r voidc lEaxses cAudted(I)n {.s.t.r}uction { g Inpu tvsco gliades tEsIn xApeductdustI(en) (s{.)t .{.r.u}..c}tion { FADD FS,FT FD ... FALU o .... In pvuotisd gEexteIncpuutets()( ){. {....}.}

r }; .. . .Inputs getInputs() {...} BEQ RS,RT ... BU p }; ....

ta };

LD RS RD ... MEM e

ST RS,RT ... MEM M

Problem: Implementation varies greatly. For example, the handling of FP registers and GP registers is significantly different.

Current solution: code generation script in Perl Existing Metaprogramming Systems

c C/C++ Macros

i

f

i c ●

e Efficient

p ● Not Language Aware S Generics: Aspect-Oriented Eiffel, Java, C# ● Very specific ● Can be Efficient applications ● Language Aware ● Still limited C++ templates

● l Efficient We want to be

a ● Language Aware s

r ●Powerful: specialization,

e HERE

v traits, expr. templates, i

n Turing machine equiv, ... U

Simple Complex Metaprogramming Contd.: Aspect-Oriented

Separation of Concerns: Validation

Synchro- nization

Logging

class A class B class C

●The programmers are advised to make the program functional even without the aspects TheA Solution: OpenJava

● The metaprogram processes the program at the source level

● Same language (Java) for both!

● Effectively, the metaprogram extends the language

Program

OpenJava Java Executable translator compiler

Metaprogram OpenJava runtime Using OpenJava

● Rename the files from *.java to *.oj – Java programs are always valid OpenJava – Add the hooks for the metaprogram in the code – Write the metaprogram

● Call ojc instead of javac – ojc will create the *.java files and call the compiler on them – ojc can also apply some of the transformations to bytecode Example: Observer Pattern class MyMenuListener implements MenuListener { void menuSelected(MenuEvent e) { /* Implementation */ }

void menuDeselected(MenuEvent e) { return; } void menuCanceled(MenuEvent e) { return; } }

● Can we automate it? class MyMenuListener implements MenuListener instantiates ObserverPattern { void menuSelected(MenuEvent e) {...} // The empty methods will be instantiated automatically } OJ Programming Model: OJClass

● The metaclass inherits from OJClass and is attached to the user's normal classes with the “instantiates” keyword

● OJClass services: – Metadata: information on the user's class – Hooks: Virtual functions that must be overridden to define the transformations

● Transformation types: – Callee side: class X instantiates ModifierClass – Caller side: on allocation, field access, method call etc. – Advanced specifiers: modifiers, syntax extension OJClass: Java Language's Object Model

● Class's introspective information: // Accessors Modifiers String getPackageName() String getSimpleName() setSimpleName OJModifier getModifiers() setModifiers OJClass getSuperclass() setSuperclass OJClass getDeclaredInterfaces() setInterfaces OJField getDeclaredFileds() addField, removeField OJMethod getDeclaredMethods() addMethod, removeMethod ● For non-classes: boolean isInterface() boolean isArray() OJClass getComponentType() // for array boolean isPrimitive() OJMethod: Object Model cont'd

● // Accessors Modifiers ● getName setName ● getModifiers setModifiers ● getReturnType setReturnType ● getParameterVars setParameterVars ● getBody setBody

● getBody/setBody operate with a StatementList, which IS an AST OJClass: Introspective Metadata

● Why not just Abstract Syntax Tree (AST)? 1. Hiding syntactical quirks: String[] a; String a[]; 2. Some elements are logically connected: Class name same as constructor name. Changing one requires changing the others. 3. Propagation of data: Access to the base class getMethods() returns the inherited methods as well 4. Ability to work with bytecodes OJClass: Hooks

● translateDefinition() – Process the whole definition of the class ● expandAllocation() – Process the new statements of the objects of the class ● expandArrayAllocation() – Same for arrays ● expandMethodCall() – Augment method calls ● expandFieldRead() / expandFieldWrite() – Custom handling for field accesses ● resolveException() – Resolve compilation errors associated with the class ● And more... ObserverPattern definition

Algorithm: Add empty class MyMenuListener implementation for the implements MenuListener instantiates ObserverPattern abstract methods public class ObserverPattern extends OJClass { void translateDefinition() { OJMethod[] m = this.getMethods(this); for (int i=0; i

● Replace all the instances of: new Glyph('c') ● with: GlyphFactory.createCharacter('c')

The framework will find all the calls of new Glyph and invoke the handler on them:

Expression expandAllocation(AllocationExpression expr, Environment env) { ExpressionList args = expr.getArguments(); return new MethodCall(this, “createCharacter”, args); } Caller Side: “...” in Java ● A function receives an array of elements – We would like to pass a free list of elements – Define the “generous” modifier to denote such functions – The metaprogram automatically checks the elements of the list and transforms it to an array ● Same as “Varargs” in Java 1.5 ● Example: class StringCollection instantiates FreeArgs { public generous void addAll(String[] args); } public class Test { void test(StringCollection strs) { Modifier implementation strs.addAll(“one”, “two”, “three”); will be explained later } } “...” in Java contd.

● Algorithm: 1. Register the “generous” modifier 2. in “translateDefinition”: record the parameter types of the functions marked with “generous”. 3. in “resolveException” on a missing method check if the missing method is due to the parameter list. If so, substitute the appropriate “generous” method. 4. in “expandMethodCall”: If the call is to a “generous” method: - generate an array initializer from the list of parameters - return the call with the array Custom Modifiers ● Function must override another one from the base class public class MyObject instantiates OverrideChecker { public overriding String toString() { return “...”; } // OK public overriding String tuString() { return “...”; } // ERROR }

● Implementation:

public class OverrideChecker extends OJClass { // Define the new modifier so that the compiler will // recognize it public static boolean isRegisteredModifier(String kw) { if (kw.equals(“overriding”)) return true; return OJClass.isRegisteredModifier(kw); } } Custom Modifiers contd. public class OverrideChecker extends OJClass { .... public void translateDefinition() { // Here we do no transformation, only checking OJMethod [] methods = getDeclaredMethods(); for (int i=0; i

● Used to pass parameters to the metaprogram

● New class or member modifiers – Designed to avoid conflict with the Java syntax – (somewhat) similar to C#'s attributes

● Callee Side (inside the class declaration): – Before the class body – Before the method – After the field variable in each field ● Caller Side (in other classes): – after the name of the class Syntax Extension: Adapter

● LL(k) syntax rules can be given. ● Parameters can be passed to the metaclass Example: interface Stack { public int size(); class VectorStack public Enumeration isEmpty(); instantiates AdapterClass public Object[] toArray(); public void push(Object o); adapts Vector in v to Stack public Object pop(); { public Object peek(); Vector v; } public void push(Object o) { v.addElement(0); } public Object pop() { ... } public Object peek() { ... } } Adapter Implementation 1. Define the syntax rule class AdapterClass ... { static SyntaxRule getDeclSuffix(String keyword) { if (keyword.equals(“adapts”)) { return new CompositeRule( new TypeNameRule(), new PrepPhraseRule(“in”, new IdentifierRule()), new PrepPhraseRule(“to”, new TypeNameRule())); } return null; } 2. For each method in the instantiated class If the method is not defined in the class, create it and delegate the implementation to the adapted object. Case Study

Re-Engineering the Java Buffer Library with OpenJava

Michael Jarrett, December 2003

JSR 51: New I/O APIs for the Java Platform. “java.nio” package in Java 1.4. Java Buffer Library

● Buffers for I/O of different basic types – Data type: Long, Byte, Char, Short, Int, Float, Double – Storage: Heap, Direct, View ● Direct buffers: can be Big or Little Endian ● View buffers: can have native or transposed byte order – Special handling for single-byte buffers – Read-only variant of each buffer

● Designed for high performance, so need specific implementation for each combination ● 72% of code is duplicated – No generics in Java at the time of writing ● Originally generated with some templating engine(!) Java Buffer Library cont'd

● Should be binary compatible with existing code – No caller-size transformations allowed ● Two approaches: 1. For each class, generate the ReadOnly variant automatically 8.5% reduction in code size, good maintainability 2. Class generators for each of Direct, Heap, View, instantiated with the data type Manual implementation for the ReadOnly variant 25% code reduction, fragile code ● Main problem: combining metaclasses Limitations & Weaknesses

● Transformations cannot be combined

– Only one metaclass per class

– Transformation does not apply to subclasses

● Code creation is still just text manipulation

● No way to return error status to the compiler

● Problems in speed, correctness, performance, documentation Conclusion

● The power of metaprogramming in your programming language – Also: OpenC++, OpenAda (!)

● Simple and powerful approach – Language support for design patterns – SafeSQL – Distributed object systems – Corba implementation Had potential, but did not fly :( – google(“OpenJava”): 24K hits – Last website update in 2003, last version dated 2002 The End

Questions ?