OpenJava – a Class-based Macro System for Java Michiaki Tatsubori, Shigeru Chiba, Marc-Olivier Killijian, Kozo Itano OOPSLA'98 Workshop on Reflective Programming in C++ and Java Why Metaprogramming ● Programming is sometimes tedious – Repetitive tasks – Error-prone tasks ● The computer is good at this kind of work! ● Idea: The programmer writes both the program and the metaprogram – Metaprogram is the program that processes the programmer's program at compilation time – Metaprogram can use metadata to guide the processing Example 1: Safe SQL Code Code class Employee { “select emp, manager” m bool filter(Query q) { a r “where emp.id > if (q.empID > g o manager.id d.manager.emplID) r AND ... p } .... ta “ .... e }; M Requires code analysis, complex logic and code generation. => Need code to generate code! Example 2: Processor Simulator Metadata Code Inputs Outs Impl FU m a class ADDInstruction { r class AddInstruction { ADD RS,RT RD ... ALU void Execute() {...} g class AddInstruction { Inpu tvso gide tEInxpeuctust(e) ({.) .{..}..} FADD FS,FT FD ... FALU o void Execute() {...} Inputs getInputs() {...} .... r }; .. .Inputs getInputs() {...} BEQ RS,RT ... BU p .... }; ta }; LD RS RD ... MEM e ST RS,RT ... MEM M Problem: Implementation varies greatly. For example, the handling of FP registers and GP registers is significantly different. Current solution: code generation script in Perl Existing Metaprogramming Systems c C/C++ Macros i f i c ● e Efficient p ● Not Language Aware S Generics: Aspect-Oriented Eiffel, Java, C# ● Very specific ● Can be Efficient applications ● Language Aware ● Still limited C++ templates ● l Efficient We want to be a ● Language Aware s r ●Powerful: specialization, e HERE v traits, expr. templates, i n Turing machine equiv, ... U Simple Complex Metaprogramming Contd.: Aspect-Oriented Separation of Concerns: Validation Synchro- nization Logging class A class B class C ●The programmers are advised to make the program functional even without the aspects TheA Solution: OpenJava ● The metaprogram processes the program at the source level ● Same language (Java) for both! ● Effectively, the metaprogram extends the language Program OpenJava Java Executable translator compiler Metaprogram OpenJava runtime Using OpenJava ● Rename the files from *.java to *.oj – Java programs are always valid OpenJava – Add the hooks for the metaprogram in the code – Write the metaprogram ● Call ojc instead of javac – ojc will create the *.java files and call the compiler on them – ojc can also apply some of the transformations to bytecode Example: Observer Pattern class MyMenuListener implements MenuListener { void menuSelected(MenuEvent e) { /* Implementation */ } void menuDeselected(MenuEvent e) { return; } void menuCanceled(MenuEvent e) { return; } } ● Can we automate it? class MyMenuListener implements MenuListener instantiates ObserverPattern { void menuSelected(MenuEvent e) {...} // The empty methods will be instantiated automatically } OJ Programming Model: OJClass ● The metaclass inherits from OJClass and is attached to the user's normal classes with the “instantiates” keyword ● OJClass services: – Metadata: information on the user's class – Hooks: Virtual functions that must be overridden to define the transformations ● Transformation types: – Callee side: class X instantiates ModifierClass – Caller side: on allocation, field access, method call etc. – Advanced specifiers: modifiers, syntax extension OJClass: Java Language's Object Model ● Class's introspective information: // Accessors Modifiers String getPackageName() String getSimpleName() setSimpleName OJModifier getModifiers() setModifiers OJClass getSuperclass() setSuperclass OJClass getDeclaredInterfaces() setInterfaces OJField getDeclaredFileds() addField, removeField OJMethod getDeclaredMethods() addMethod, removeMethod ● For non-classes: boolean isInterface() boolean isArray() OJClass getComponentType() // for array boolean isPrimitive() OJMethod: Object Model cont'd ● // Accessors Modifiers ● getName setName ● getModifiers setModifiers ● getReturnType setReturnType ● getParameterVars setParameterVars ● getBody setBody ● getBody/setBody operate with a StatementList, which IS an AST OJClass: Introspective Metadata ● Why not just Abstract Syntax Tree (AST)? 1. Hiding syntactical quirks: String[] a; String a[]; 2. Some elements are logically connected: Class name same as constructor name. Changing one requires changing the others. 3. Propagation of data: Access to the base class getMethods() returns the inherited methods as well 4. Ability to work with bytecodes OJClass: Hooks ● translateDefinition() – Process the whole definition of the class ● expandAllocation() – Process the new statements of the objects of the class ● expandArrayAllocation() – Same for arrays ● expandMethodCall() – Augment method calls ● expandFieldRead() / expandFieldWrite() – Custom handling for field accesses ● resolveException() – Resolve compilation errors associated with the class ● And more... ObserverPattern definition Algorithm: Add empty class MyMenuListener implementation for the implements MenuListener instantiates ObserverPattern abstract methods public class ObserverPattern extends OJClass { void translateDefinition() { OJMethod[] m = this.getMethods(this); for (int i=0; i<m.length; ++i) { OJModifier modif = m[j].getModifiers(); if (modif.isAbstract()) { OJMethod n = new OJMethod(this, ... makeStatementList(“return;”)); this.addMethod(n); } Caller Side: Flyweight ● Replace all the instances of: new Glyph('c') ● with: GlyphFactory.createCharacter('c') The framework will find all the calls of new Glyph and invoke the handler on them: Expression expandAllocation(AllocationExpression expr, Environment env) { ExpressionList args = expr.getArguments(); return new MethodCall(this, “createCharacter”, args); } Caller Side: “...” in Java ● A function receives an array of elements – We would like to pass a free list of elements – Define the “generous” modifier to denote such functions – The metaprogram automatically checks the elements of the list and transforms it to an array ● Same as “Varargs” in Java 1.5 ● Example: class StringCollection instantiates FreeArgs { public generous void addAll(String[] args); } public class Test { void test(StringCollection strs) { Modifier implementation strs.addAll(“one”, “two”, “three”); will be explained later } } “...” in Java contd. ● Algorithm: 1. Register the “generous” modifier 2. in “translateDefinition”: record the parameter types of the functions marked with “generous”. 3. in “resolveException” on a missing method check if the missing method is due to the parameter list. If so, substitute the appropriate “generous” method. 4. in “expandMethodCall”: If the call is to a “generous” method: - generate an array initializer from the list of parameters - return the call with the array Custom Modifiers ● Function must override another one from the base class public class MyObject instantiates OverrideChecker { public overriding String toString() { return “...”; } // OK public overriding String tuString() { return “...”; } // ERROR } ● Implementation: public class OverrideChecker extends OJClass { // Define the new modifier so that the compiler will // recognize it public static boolean isRegisteredModifier(String kw) { if (kw.equals(“overriding”)) return true; return OJClass.isRegisteredModifier(kw); } } Custom Modifiers contd. public class OverrideChecker extends OJClass { .... public void translateDefinition() { // Here we do no transformation, only checking OJMethod [] methods = getDeclaredMethods(); for (int i=0; i<methods.length; ++i) { if (!methods[i].getModifiers().has(“overriding”)) continue; String name = methods[i].getName(); try { getSuperclass().getMethod(name, ...); } catch (NoSuchMemberException) { // Notify the programmer of the problem. } } } Syntax Extension ● Used to pass parameters to the metaprogram ● New class or member modifiers – Designed to avoid conflict with the Java syntax – (somewhat) similar to C#'s attributes ● Callee Side (inside the class declaration): – Before the class body – Before the method – After the field variable in each field ● Caller Side (in other classes): – after the name of the class Syntax Extension: Adapter ● LL(k) syntax rules can be given. ● Parameters can be passed to the metaclass Example: interface Stack { public int size(); class VectorStack public Enumeration isEmpty(); instantiates AdapterClass public Object[] toArray(); public void push(Object o); adapts Vector in v to Stack public Object pop(); { public Object peek(); Vector v; } public void push(Object o) { v.addElement(0); } public Object pop() { ... } public Object peek() { ... } } Adapter Implementation 1. Define the syntax rule class AdapterClass ... { static SyntaxRule getDeclSuffix(String keyword) { if (keyword.equals(“adapts”)) { return new CompositeRule( new TypeNameRule(), new PrepPhraseRule(“in”, new IdentifierRule()), new PrepPhraseRule(“to”, new TypeNameRule())); } return null; } 2. For each method in the instantiated class If the method is not defined in the class, create it and delegate the implementation to the adapted object. Case Study Re-Engineering the Java Buffer Library with OpenJava Michael Jarrett, December 2003 JSR 51: New I/O APIs for the Java Platform. “java.nio” package in Java 1.4. Java Buffer Library ● Buffers for I/O of different basic types – Data type: Long, Byte, Char, Short, Int, Float, Double – Storage: Heap, Direct, View ● Direct buffers: can be Big or Little Endian ● View buffers: can have native or transposed