PROC. OF THE 8th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2015) 39
Garbage Collection in JyNI – How to bridge Mark/Sweep and Reference Counting GC
Stefan Richthofer∗†
!
Abstract—Jython is a Java-based Python implementation and the most seam- fact that usually ties this kind of code to CPython. Developing less way to integrate Python and Java. It achieves high efficiency by compiling and maintaining such bindings is usually a difficult and error- Python code to Java bytecode and thus letting Java’s JIT optimize it – an prone task. One major goal of the JyNI-project is to let Python approach that enables Python code to call Java functions or to subclass Java and Java – with the help of [JYTHON] – share their pools of classes. It enables Python code to leverage Java’s multithreading features and utilizes Java’s built-in garbage collection (GC). However, it currently does not language bindings, vastly enriching both ecosystems. support CPython’s C-API and thus does not support native extensions like While Jython already enables Python code to access Java NumPy and SciPy. Since most scientific code depends on such extensions, it frameworks and also native JNI-based (Java Native Interface) is not runnable with Jython. C extensions, it currently locks out all extensions that use Jython Native Interface (JyNI) is a compatibility layer that aims to provide CPython’s native API [C-API]. Remember that this does CPython’s native C extension API on top of Jython. JyNI is implemented using not only affect the actual C extensions, but also all Python the Java Native Interface (JNI) and its native part is designed to be binary compatible with existing extension builds. This means Jython can import the frameworks that have a – maybe single, subtle – dependency original C extensions, i.e. the same .dll- or .so-files that CPython would use. on such an extension. Dependencies can include: For various reasons, implementing CPython’s C-API is not an easy task. • Libraries like NumPy that are written directly in terms Just to name a few issues – it offers macros to access CPython internals, uses a global interpreter lock (GIL) in contrast to Jython and lets extensions of the C-API. These libraries, which in turn link native perform reference-counting-based GC, which is incompatible with Java’s GC- libraries like BLAS, are widely used in the Python approach. For each of the arising issues, JyNI proposes a feasible solution; most ecosystem, especially in scientific code. remarkably it emulates CPython’s reference-counting GC on top of Java’s mark- • [CYTHON] is a popular tool to build optimized C code and-sweep-based approach (taking care of adjacent concepts like finalizers from Python source that has been annotated with types and weak references and their interference with Jython). (Note that there are and other declaration, using the C-API to link. vague considerations around to switch to mark-and-sweep-based GC in a future • CPython too; cf. [PY3_PL15]. So this algorithm might one day be even relevant The [CTYPES] and [CFFI] modules, comparable to to CPython in terms of running legacy modules.) [JNA] and [JNR] in the Java-world respectively, are other This work’s main purpose is to describe the algorithm JyNI uses to support popular means of providing support for C bindings, also GC in detail, also covering weak references and testing native memory man- all written to use the C-API. agement. Beside this we give a comprehension of JyNI’s general design and • [SWIG], [PYREX] (from which Cython was derived), briefly describe how it deals with CPython’s GIL. Finally we provide runnable Boost.Python ([BOOSTPY]) and [SIP] are further tools code examples, e.g. a demonstration of JyNI’s support for the ctypes extension that create extensions using the C-API. and a first experimental import of NumPy.
arXiv:1607.00825v1 [cs.PL] 1 Jul 2016 [JyNI] (Jython Native Interface) is going to improve this Index Terms—Jython, Java, Python, CPython, extensions, integration, JNI, situation. It is a compatibility layer that implements CPython’s native, NumPy, C-API, SciPy, GC C-API on top of JNI and Jython. This way it enables Jython to load native CPython extensions and use them the same way 1 INTRODUCTION as one would do in CPython. To leverage this functionality, As interpreter based languages, Python and Java both depend no modification to Python code or C extension source code on native language bindings/extensions in many scenarios. is required – one just needs to add JyNI.jar to Jython’s Especially scientific code mostly relies on NumPy or native classpath (along with its binary libraries). That means JyNI is interfaces to some computation- or control-framework that binary compatible with existing builds of CPython extensions. connects Python to problem-specific hardware or libraries – a Developing JyNI is no trivial task, neither is it completed yet. Main reason for this is Python’s rather complex C-API * Corresponding author: [email protected] that allows to access internal structures, methods and memory † Institute for Neural Computation, Ruhr-Universität Bochum positions directly or via C preprocessor macros (in some sense Copyright ○c 2015 Stefan Richthofer. This is an open-access article dis- CPython simply exposes its own internal API via a set of tributed under the terms of the Creative Commons Attribution Li- public headers). Existing extensions frequently do make use cense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. of this, so it is not a purely academical concern. Concepts like http://creativecommons.org/licenses/by/3.0/ Python’s global interpreter lock (GIL), exception handling and 40 PROC. OF THE 8th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2015) the buffer protocol are further aspects that complicate writing None of the named approaches reached a sufficient level of JyNI. [PMB_PL15] mentions the same issues from [PyPy]’s functionality/compatibility, at least not for current language perspective and confirms the difficulty of providing CPython’s versions (some of them used to work to some extend, but native API. became unmaintained). In the Python ecosystem the C ex- By far the most complex issue overall – and main focus tension API has been an ongoing issue since its beginning. of this work – is garbage collection. Unlike JNI, CPython PyPy famously has been encouraging developers to favor CFFI offers C-level access to its garbage collector (GC) and ex- over C extension API, as it is the only existing approach tensions can use it to manage their memory. Note that in that has been designed to be well portable to other Python contrast to Java’s mark-and-sweep-based GC, CPython’s GC implementations. However, even if this effort would work out, uses reference-counting and performs reference-cycle-search. there would be so many legacy extensions around that a serious Adopting the original CPython-GC for native extensions is move to CFFI won’t be done in foreseeable future2. no feasible solution in JyNI-context as pure Java-objects can For some of these projects JyNI’s GC-approach might be a become part of reference-cycles that would be untraceable and relevant inspiration, as they face the same problem if it comes cause immortal trash. Section 3.1 describes this issue in detail. to native extensions. There are even vague considerations for While there are conceptual solutions for all mentioned CPython to switch to mark-and-sweep-based GC one day to issues, JyNI does not yet implement the complete C-API and enable a GIL-free version (c.f. [PY3_PL15]). Background here currently just works as a proof of concept. It can already is the fact that reference-counting-based garbage collection is load the ctypes extension and perform some tasks with it; the main reason why CPython needs a GIL: Current reference e.g. it can successfully run several tests of a slightly modified counters are not atomic and switching to atomic reference PyOpenGL ([Py_OGL]) version (some very long methods had counters yields insufficient performance. In context of a mark- to be split to comply with limited method-length supported by and-sweep-based garbage collection in a future CPython the the JVM, see [Jy_OGL]). We are working to complete ctypes JyNI GC-approach could potentially be adopted to support support as soon as possible, since many Python libraries, e.g. legacy extensions and provide a smooth migration path. for graphics (PyOpenGL) and 3D-plotting etc., have only a single native dependency on it. NumPy and SciPy are other important extensions we plan to support with priority. As of 2 IMPLEMENTATION version alpha.4 JyNI can import the latest NumPy repository version (upcoming NumPy 1.12) and perform some very basic In order to bridge Jython’s and CPython’s concepts of PyOb- operations with it (cf. section 5.3); however a full support is jects, we apply three different techniques, depending on the still in unknown distance (cf. section6). PyObject’s implementation details.
1.1 Overview PyObject* PyObject* JyNI’s basic functionality has been described in [JyNI_EP13]. PyObject* After giving a short comprehension in section2 we will focus on garbage collection in section3. For usage examples and a demonstration-guide also see [JyNI_EP13]. Section4 focuses jobject jobject on the difficulties regarding weak references and section5 jobject discusses some demonstration examples. Fig. 1: Approaches to bridge PyObjects. Left: Native PyObject wraps Java. Center: Java-PyObject wraps native one. Right: Objects are 1.2 Related Work mirrored. There have been similar efforts in other contexts. • [JEP] and [JPY] can bridge Java and Python by em- The basic approach is to back the C-API of PyObject by a bedding the CPython interpreter. However, none of these Java-PyObject via JNI. This would avoid data synchronization approaches aims for integration with Jython. In contrast issues, but is only feasible if there are matching counterparts to that, JyNI is entirely based on Jython and its runtime. of the PyObject type in Jython and CPython (fig.1, left). • Ironclad ([ICLD]) is a JyNI-equivalent approach for Iron- For CPython-specific types we can do it the other way round Python ([IRPY]). (fig.1, center). Another problem is that CPython API defines • PyMetabiosis ([PMB]) provides C extension support in macros in public headers that access PyObjects’ internal data. PyPy to some extent by embedding the CPython inter- To deal with these, we sometimes have to mirror the object preter. So its approach is comparable to [JEP] and [JPY]. (fig.1, right). This might involve data synchronization issues, • [CPYEXT] refers to [PyPy]’s in-house (incomplete) C but luckily macros mostly exist for immutable types, so initial extension API support. The approach differs from [JyNI] synchronization is sufficient. [JyNI_EP13] describes this in by requiring recompilation and sometimes adjustments of more detail. the extensions using PyPy-specific headers1. 2. Our plan is to support the CPython-variant of CFFI in JyNI as an 1. This yields advantages and disadvantages compared to JyNI; discussing alternative to the ideal approach of a direct port. Creating a true Jython version these is out of scope for this work. of CFFI would be a distinct project and was partly done based on [JNR]/JFFI. GARBAGE COLLECTION IN JYNI – HOW TO BRIDGE MARK/SWEEP AND REFERENCE COUNTING GC 41
peer with ReferenceQueue 2.1 Global interpreter lock after GC native handle As mentioned before, CPython needs a GIL, because poll its reference-counting-based GC uses non-atomic refer- weak reference ence counters. That means that CPython is entirely Java/Jython also needs handle single-threaded in its usual operation mode. A na-
JNI/C inc. refcount tive extension can explicitly release the GIL by in- serting the macros Py_BEGIN_ALLOW_THREADS and dec. refcount Py_END_ALLOW_THREADS to deal with multiple threads native backend and related things like input events (e.g. Tkinter needs this). In Fig. 2: Ordinary JNI memory management the potentially multithreaded code between these macros it is the extension’s own responsibility to refrain from non-thread- safe operations like incrementing or decrementing reference Figure2 sketches the following procedure: counters. This can be error-prone and challenging as the • a java.lang.ref.WeakReference is used to track extension must ensure this also for eventually called external the peer methods. • actually we store a copy of the peer’s native handle in a Jython on the other hand has no GIL and is fully multi- subclass of java.lang.ref.WeakReference threaded based on Java’s threading architecture. This does not • a java.lang.ref.ReferenceQueue is registered mean multithreading would be trivial – one still has to care for with the weak reference concurrency issues and thread synchronization, but the whole • after every run, Java-GC automatically adds cleared weak machinery Java came up with for this topic is available to deal references to such a queue if one is registered (this is with it. Java’s variant of Python’s weak reference callbacks) From JyNI’s perspective this is a difficult situation. On one • we poll from the reference queue and clean up the hand we want to avoid regressions on Jython-site, especially corresponding native resource regarding an important feature like GIL-freeness. On the • since other native objects might need the resource, we other hand, native C extensions might rely on CPython’s don’t call free, but instead perform reference counting GIL. So as a compromise JyNI provides a GIL for native So far this would work, but JyNI also needs the opposite site that is acquired by any thread that enters native code. scenario with a native peer backed by a Java-object (figure3). On returning to Java code, i.e. finishing the native method call, the JyNI-GIL is released. Note that re-entering Java- Java backend site by doing a Java call from a native method would not release the GIL. In case it is desired to release the GIL JNI: for such a re-entering of Java-site or in some other situa- Java/Jython GlobalRef tion, JyNI also supports Py_BEGIN_ALLOW_THREADS and JNI/C Py_END_ALLOW_THREADS from CPython. This architecture implies that multiple threads can exist on Java-site, while native peer only one thread can exist on native site at the same time (unless allow-threads macros are used). When combining Fig. 3: A native peer backed by a Java-object multithreaded Jython code with JyNI it is the developer’s responsibility to avoid issues that might arise from this design. To prevent Java-GC from destroying the Java-backend while it is in use, JNI offers the concept of global references – JNI- GlobalRef-objects. However, native code must explicitly 3 GARBAGE COLLECTION create and release such global references. During the lifetime While there are standard approaches for memory management of a native global reference the Java-site referent is immortal. in context of JNI, none of these is applicable to JyNI. In this Now consider the referent would hold further references to section we sketch the default approaches, illustrate why they other Java-objects. The reference chain could at some point fail and finally provide a feasible solution. include an object that is a peer like shown in figure2. This peer would keep alive a native object by holding a reference- 3.1 Why is Garbage Collection an issue? increment on it. If the native object also holds reference- Consider a typical JNI-scenario where a native object is increments of other native objects this can create a pathological accessed from Java. Usually one would have a Java-object reference cycle like illustrated in figure4. (a “peer”) that stores the native memory address of the C- This kind of cycle cannot be cleared by Java-GC as the object (i.e. a pointer to it) in a long-field. The naive approach GlobalRef prevents it. Native reference cycle search like to do memory management would be a finalize-method known from CPython could not resolve the cycle either, in the peer-class. This finalizer would then trigger a native because it cannot be traced through Java-site. For debug- free-call on the stored memory-handle. However, finalizers ging purposes we actually added a traverseproc-mechanism are considered bad style in Java as they impact GC-efficiency. to Jython that would allow to trace references through Java- The recommended approach for this scenario is based on weak site, but to clear such a cycle in general just tracing Java-site references and a reference-queue (c.f. [JREF]). references is not sufficient; Java-site reference counting would 42 PROC. OF THE 8th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2015)
Java backend peer JyGCHead
JNI:
Java/Jython GlobalRef immortal inc. ref
JNI/C Java/Jython inc. ref inc. ref inc. ref inc. ref
JNI/C inc. ref native backend native peer JNI: WeakGlobalRef Fig. 4: A pathological reference cycle immortal PyObject be required. This in turn would Jython require to have a GIL, Fig. 6: clearing unreachable objects which would be an unacceptable regression.
Java’s GC runs before this change was mirrored to Java-site. 3.2 How JyNI solves it (basic approach) In that case two types of errors could normally happen: To solve this issue, JyNI explores the native reference graph 1) Objects might be deleted that are still in use using CPython’s traverseproc mechanism. This is a mecha- 2) Objects that are not in use any more persist nism PyObjects must implement in order to be traceable by The design applied in JyNI makes sure that only the second CPython’s garbage collector, i.e. by the code that searches type of error can happen and this only temporarily, i.e. objects for reference cycles. Basically a PyObject exposes its ref- might persist for an additional GC-cycle or two, but not erences to other objects this way. While JyNI explores the forever. To make sure that the first kind of error cannot happen, native reference graph, it mirrors it on Java-site using some we check a to-be-deleted native reference subgraph for inner minimalistic head-objects (JyNIGCHead s); see figure5. consistency before actually deleting it. Note that with this design, also Java-objects, especially Jython- PyObjects can participate in the reference graph and keep parts JyGCHead of it alive. The kind of object that needed a JNI-GlobalRef in figure4, can now be tracked by a JNI- WeakGlobalRef while it is kept alive by the mirrored reference graph on Java- immortal site as figure5 illustrates. Java/Jython
JNI/C inc. ref JyGCHead Java-Object immortal immortal PyObject
Java/Jython inc. ref inc. ref Fig. 7: graph must be checked for inner consistency (GC ran before
JNI/C inc. ref orange connection was mirrored to Java-site)
JNI: WeakGlobalRef If not all native reference counts are explainable within
immortal this subgraph (c.f. figure7), we redo the exploration of PyObject participating PyObjects and update the mirrored graph on Java- Fig. 5: reflected native reference graph site.
JyGCHead If a part of the (native) reference-graph becomes unreach- able (figure6), this is reflected (asynchronously) on Java-site.
On its next run, Java-GC will collect this subgraph, causing immortal weak references to detect deleted objects and then release
Java/Jython inc. ref native references. inc. ref
JNI/C inc. ref
3.3 How JyNI solves it (hard case)
The fact that the reference-graph is mirrored asynchronously immortal PyObject can lead to bad situations. While JyNI features API that allows C code to report changes of the graph, we cannot enforce Fig. 8: recreated graph third-party-written native extensions to report such changes. However, we made sure that all built-in types instantaneously While we can easily recreate the GC-heads, there might be send updates to Java-site on modification. PyObjects that were weakly reachable from native site and Now consider that a native extension changes the reference were swept by Java-GC. In order to restore such objects, we graph silently (e.g. using macro access to a PyObject) and must perform a resurrection (c.f. figure9). GARBAGE COLLECTION IN JYNI – HOW TO BRIDGE MARK/SWEEP AND REFERENCE COUNTING GC 43
JyGCHead resurrect DemoExtension.argCountToString(lst) del lst print "Leaks before GC:"
immortal monitor.listLeaks() System.gc()
Java/Jython inc. ref inc. ref time.sleep(2) print "Leaks after GC:" JNI/C inc. ref monitor.listLeaks() It creates a reference cycle, passes it to a native function and
immortal deletes it afterwards. By passing it to native code, a native PyObject counterpart of lst was created, which cannot be cleared Fig. 9: resurrected Java-backend without some garbage collection (also in CPython it would need the reference cycle searching GC). We list the leaks before calling Java’s GC and after running it. The output is as The term “object-resurrection” refers to a situation where an follows: object was garbage-collected, but has a finalizer that restores Leaks before GC: a strong reference to it. Note that while resurrection is not Current native leaks: recommended – actually the possibility of a resurrection is 140640457447208_GC (list) #2: "[([...],), ’test’]"_j *38 the main reason why finalizers are not recommended – it is 140640457457768_GC (tuple) #1: a legal operation. So certain GC-heads need to be able to "(([([...],), ’test’],),)"_j *38 resurrect an underlying Jython-PyObject and thus must have a 140640457461832 (str) #2: "test"_j *38 finalizer. Since only certain objects can be subject to a silent 140640457457856_GC (tuple) #3: "([([...],), ’test’],)"_j *38 reference-graph modification, it is sufficient to let only GC- Leaks after GC: heads attached to these objects implement finalizers – we use no leaks recorded finalizers only where really needed. We can see that it lists some leaks before running Java’s GC. Each line consists of the native memory position, the 3.4 Testing native garbage collection type (in parentheses), the current native reference count in- Since the proposed garbage collection algorithm is rather dicated by #, a string representation and the creation time involved, it is crucial to have a good way to test it. To achieve indicated by * in milliseconds after initialization of the this we developed a monitoring concept that is capable of JyReferenceMonitor class. The postfix _GC means that tracking native allocations, finalizations, re- and deallocations. the object is subject to garbage collection, i.e. it can hold The class JyNI.JyReferenceMonitor can – if native references to other objects and thus participate in cycles. monitoring is enabled – list at any time all natively allocated Objects without _GC will be directly freed when the reference objects, their reference counts, timestamps for allocation, counter drops to zero. The postfix _j of the string represen- finalization, re- and deallocations and the corresponding code tation means that it was generated by Jython rather than by positions (file and line-number) that performed the memory native code. We close this section by discussing the observed operations. Unless explicitly cleared, it can also provide reference counts: history of these actions. The method listLeaks() lists • The list-object has one reference increment from its all currently allocated native objects (actually these are not JyGCHead and the other from the tuple at the bottom necessarily leaks, if the method is not called at the end of the output. of a program or test). While listLeaks() is useful for • The first-listed tuple is the argument-tuple and only debugging, getCurrentNativeLeaks() provides a list referenced by its JyGCHead. that is ideal for unit testing. E.g. one can assert that no objects • The string is referenced by its JyGCHead and the list. are leaked: • The tuple at the bottom is referenced by its JyGCHead, from JyNI import JyReferenceMonitor as monitor by the list and by the argument-tuple. #... self.assertEqual( len(monitor.getCurrentNativeLeaks()),0) The native counterpart of JyNI.JyReferenceMonitor 4 WEAK REFERENCES is JyRefMonitor.c. Its header defines the JyNIDebug Supporting the PyWeakRef built-in type in JyNI is not as macro family, wich we insert into C code wherever memory complicated as garbage collection, but still a notably involved operations occur (mainly in obmalloc.c and various inlined task. This is mainly due to consistency requirements that are allocations in stringobject.c, intobject.c etc.). not trivial to fulfill. Consider the following demonstration code: • If a Jython weakref-object is handed to native site, this import time from java.lang import System shall convert to a CPython weakref-object and vice versa. from JyNI import JyReferenceMonitor as monitor • If native code evaluates a native weakref, it shall return import DemoExtension exactly the same referent-PyObject that would have been JyNI.JyRefMonitor_setMemDebugFlags(1) lst=([0,"test"],) created if the Java-pendant (if one exists) was evaluated l[0][0]= lst and the result was handed to native site; also vice versa. 44 PROC. OF THE 8th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2015)
PyWeakReferences • If a Jython weak reference is cleared, its native pendant C-PyObject linked list double- shall be cleared either. Still, none of them shall be cleared referent as long as its referent is still alive. • This implies that even if a Jython referent-PyObject was deleted (can happen in mirror-case) Jython weakref- Fig. 11: CPython’s concept for weak references objects stay alive as long as the native pendant of the referent is alive. If evaluated, such a Jython weakref- object retrieves the Jython referent by converting the Figure 11 shows that – like in Jython – referents track weak native referent. references pointing to them; in this case references are stored • An obvious requirement is that this shall of course work in a double-linked list, allowing to iterate them for callback- without keeping the referents alive or creating some kind processing. of memory leak. JyNI’s delicate GC mechanism must be taken into account to fulfill the named requirements in this context. 4.3 Weak References in JyNI In the following, we explain JyNI’s solution to this issue. via weak, static map PyObject We start by explaining the weakref-concepts of Jython and Java CPython, completing this section by describing how JyNI WeakReference Java-referent AbstractRef combines them to a consistent solution. Note that CPython’s Native weakref-module actually introduces three built-in types: GlobalRef Mirror-mode • _PyWeakref_RefType (“weakref”) Java/Jython inc. ref
• _PyWeakref_ProxyType (“weakproxy”) JNI/C PyWeakReferences various cases • _PyWeakref_CallableProxyType linked list double-
(“weakcallableproxy”) native referent
Fig. 12: JyNI’s concept for weak references 4.1 Weak References in Jython In Jython the package org.python.modules._weakref JyNI’s weak reference support is grounded on CPython’s contains the classes that implement weak reference support. approach on native site and Jython’s approach on Java-site. • ReferenceType implements the “weakref”-built-in However, the actual effort is to bridge these approaches in • ProxyType implements the “weakproxy”-built-in a consistent way. To fulfill the requirement for consistent • CallableProxyType implements the clear-status, we establish a “Java-referent dies first”-policy. “weakcallableproxy”-built-in Instead of an ordinary GlobalRef, JyNI uses a subclass All of them extend AbstractReference, which in turn called NativeGlobalRef. This subclass holds a reference- extends PyObject. increment for the native counterpart of its referent. This en- sures that the native referent cannot die as long as Jython weak via weak, static map PyObject references exist (see figure 12). Otherwise, native weak refer- Java ences might be cleared earlier than their Jython-pendants. Note WeakReference AbstractRef that the native ref-increment held by NativeGlobalRef Jython PyObject cannot create a reference-cycle, because it is not reflected by referent via JyAttribute a JyNIGCHead as seen in figure5. Also, the consistency- GlobalRef check shown in figure7 takes this ref-increment into account, Fig. 10: Jython’s concept for weak references i.e. tracks ref-increments coming from NativeGlobalRef s separately. As figure 10 illustrates, Jython creates only one via weak, static map Java-style weak reference per referent. This is cre- PyObject ated in form of a GlobalRef-object, which extends java.lang.ref.WeakReference. It stores all Jython AbstractRef Native weak references pointing to it in a static, weak-referencing GlobalRef map. This is needed to process potential callbacks when the Java/Jython Mirror-mode reference is cleared. Once created, a GlobalRef is tied to its referent, kept alive by it and is reused throughout the referent’s JNI/C neutral ref-count PyWeakReferences linked list double- lifetime. Finally, AbstractReference-subclasses refer to the GlobalRef corresponding to their actual referent. native referent
Fig. 13: JyNI weak reference after Java-referent was collected 4.2 Weak References in CPython In CPython, each weakref-type simply contains a reference to If the Jython-referent and its native pendant are handled in its referent without increasing reference count. mirror-mode, it can happen that the Java-referent is garbage- GARBAGE COLLECTION IN JYNI – HOW TO BRIDGE MARK/SWEEP AND REFERENCE COUNTING GC 45 collected while the native one persists. As soon as the Jython- referent is collected, its NativeGlobalRef releases the native reference-increment (see figure 13). Still, it will not be cleared and process callbacks, before also the native referent dies. Until then, NativeGlobalRef continues to be valid – it implements its get-method such that if the Jython-referent is not available, it is recreated from the native referent. As long as such a retrieved referent is alive on Java-site, the situation in figure 12 is restored. Fig. 14: Tkinter demonstration by Java code. Note that the class JyNI.TestTk is executed rather than org.python.util.jython. 5 EXAMPLES The code-samples in this section are runnable with Jython 2.7.1 and JyNI 2.7-alpha.3 or newer (alpha.4 for NumPy- public static interface Entry { import). public void pack(); }
5.1 Using Tkinter from Java public static interface Tk { public void mainloop(); In [JyNI_EP13] we demonstrated a minimalistic Tkinter exam- public void destroy(); ple program that used the original Tkinter binary bundled with } CPython. Here we demonstrate how the same functionality public static interface StringVar { can be achieved from Java code. This confirms the usability public String get(); of Python libraries from Java via Jython and JyNI. While public void set(String text); } the main magic happens in Jython, it is not completely self- evident that this is also possible through JyNI and required We define the methods backing the button-actions as static some internal improvements to work. Remember the Tkinter- methods with a special Python-compliant signature: program from [JyNI_EP13]: static Tk root; import sys static StringVar txt; #Include native Tkinter: sys.path.append(’/usr/lib/python2.7/lib-dynload’) public static void printText(PyObject[] args, sys.path.append(’/usr/lib/python2.7/lib-tk’) String[] kws){ System.out.println(txt.get()); from Tkinter import * } root= Tk() txt= StringVar() public static void printTimeStamp(PyObject[] args, txt.set("Hello World!") String[] kws){ System.out.println("System.currentTimeMillis: " def print_text(): + System.currentTimeMillis()); print txt.get() } def print_time_stamp(): public static void destroyRoot(PyObject[] args, from java.lang import System String[] kws){ print "System.currentTimeMillis:" root.destroy(); +str(System.currentTimeMillis()) }
Label(root, On top of this a rather Java-like main-method can be im- text="Welcome to JyNI Tkinter-Demo!").pack() plemented. Note that constructing objects is still somewhat Entry(root, textvariable=txt).pack() unhandy, as keywords must be declared in a string-array and Button(root, text="print text", command=print_text).pack() explicitly passed to Jython. Calling methods on objects then Button(root, text="print timestamp", works like ordinary Java code and is even type-safe based on command=print_time_stamp).pack() the declared interfaces. Button(root, text="Quit", command=root.destroy).pack() public static void main(String[] args){ root.mainloop() PySystemState pystate= Py.getSystemState(); pystate.path.add( To translate the program to Java, we must provide type- "/usr/lib/python2.7/lib-dynload"); information via interfaces: pystate.path.add("/usr/lib/python2.7/lib-tk"); PyModule tkModule=(PyModule) import org.python.core.*; imp.importName("Tkinter", true); root= tkModule.newJ(Tk.class); public static interface Label { txt= tkModule.newJ(StringVar.class); public void pack(); txt.set("Hello World!"); } Label lab= tkModule.newJ(Label.class, public static interface Button { new String[]{"text"}, root, public void pack(); "Welcome to JyNI Tkinter-Demo!"); } lab.pack(); 46 PROC. OF THE 8th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2015)
Entry entry= tkModule.newJ(Entry.class, “java” in Jython, breaking the original logic. See [JyNI_GSoC] new String[]{"textvariable"}, root, txt); for details. entry.pack();
String[] kw_txt_cmd={"text", "command"}; 5.3 Experimental NumPy-import Button buttonPrint= tkModule.newJ(Button.class, kw_txt_cmd, root, "print text", As of JyNI alpha.4 it is possible to import NumPy 1.12 Py.newJavaFunc(TestTk.class, "printText")); (current repository version, not yet released) and perform some buttonPrint.pack(); very basic operations: import sys Button buttonTimestamp= tkModule.newJ( sys.path.append(’path_to_numpy1.12_pre-release’) Button.class, kw_txt_cmd, import numpy as np root, "print timestamp", Py.newJavaFunc(TestTk.class, a= np.array([2,5,7]) "printTimeStamp")); buttonTimestamp.pack(); print a print 3 a Button buttonQuit= tkModule.newJ(Button.class, * print np.outer(a, a) kw_txt_cmd, root, "Quit", Py.newJavaFunc(TestTk.class, "destroyRoot")); This example yields the expected output: buttonQuit.pack(); [257] [6 15 21] root.mainloop(); [[4 10 14] } [10 25 35] [14 35 49]] 5.2 Using native ctypes E.g. operations involving floating point numbers fail as of As of version alpha.3 JyNI has experimental support for this writing, so a usable support is still in some distance. [CTYPES]. The following code provides a minimalistic ex- Considering its vast complexity, mastering NumPy’s import- ample that uses Java- and C-API. Via an std-lib C-call we script numpy/__init__.py is yet a crucial milestone on this obtain system time and print it using Java console. front. import sys Major challenges here were dependencies on several other sys.path.append(’/usr/lib/python2.7/lib-dynload’) extensions. E.g. NumPy depends on ctypes and datetime (i.e. on datetime’s C-API, rendering the Jython-variant of datetime import ctypes insufficient) – see ctypes example in 5.2 and datetime example from java.lang import System in [JyNI_EP13]. libc= ctypes.CDLL(’libc.so.6’) NumPy also contains Cython-generated code, which results print libc in huge and hard to read C source-files. E.g. mtrand.c from print libc.time System.out.println(’Timestamp:’+str(libc.time(0))) NumPy’s random module contains more than 40.000 lines of code, making it a true challenge to debug. In this sense The output is as follows: the feasible NumPy-import also demonstrates a basic Cython-
[JyNI_EP13] Stefan Richthofer, JyNI - Using native CPython-Extensions in Jython, Proceedings of the 6th European Conference on Python in Science (EuroSciPy 2013), http://arxiv.org/abs/1404.6390, 2014-05-01, Web. 2016-07-01 [JYTHON] Python Software Foundation, Corporation for National Re- search Initiatives, Jython: Python for the Java Platform, http: //www.jython.org, 2015-09-11, Web. 2016-07-01 [C-API] Python Software Foundation, Python/C API Reference Manual, http://docs.python.org/2/c-api, Web. 2016-07-01 [CYTHON] Robert Bradshaw, Stefan Behnel, Dag Seljebotn, Greg Ewing et al., Cython, http://cython.org, 2015-10-10, Web. 2016-07-01 [CTYPES] Thomas Heller, ctypes, http://starship.python.net/crew/theller/ ctypes, Web. 2016-07-01 [CFFI] Armin Rigo, Maciej Fijalkowski, CFFI, http://cffi.readthedocs. org/en/latest, 2015, Web. 2016-07-01 [JNA] Todd Fast, Timothy Wall, Liang Chen et al., Java Native Access, https://github.com/java-native-access/jna, Web. 2016-07-01 [JNR] Charles Nutter, Thomas Enebo, Nick Sieger, Java Native Run- time, https://github.com/jnr, 2015, Web. 2016-07-01 [SWIG] Dave Beazley, William Fulton et al., SWIG, http://www.swig. org, Web. 2016-07-01 [PYREX] Greg Ewing, Pyrex, http://www.cosc.canterbury.ac.nz/greg. ewing/python/Pyrex, Web. 2016-07-01 [BOOSTPY] Dave Abrahams, Boost.Python, http://www.boost.org/doc/libs/ 1_59_0/libs/python/doc/index.html, 2003, Web. 2016-07-01 [SIP] Phil Thompson, Reverbank Computing, SIP https: //riverbankcomputing.com/software/sip/intro, 2015, Web. 2016-07-01 [PMB] Romain Guillebert, PyMetabiosis, https://github.com/ rguillebert/pymetabiosis, Web. 2016-07-01 [PMB_PL15] Romain Guillebert (write-up by Jake Edge), PyMBbiosis, Python Language Summit 2015, PyCon 2015, LWN.net, https: //lwn.net/Articles/641021, Web. 2016-07-01 [PyPy] Armin Rigo, Samuele Pedroni, Christian Tismer, Holger Krekel et al., PyPy, http://pypy.org, 2015-06-01, Web. 2016-07-01 [Py_OGL] Mike C. Fletcher, PyOpenGL, http://pyopengl.sourceforge.net, 2015-02-13, Web. 2016-07-01 [Jy_OGL] Stefan Richthofer, JyOpenGL, https://github.com/Stewori/ JyOpenGL, 2016-04-10, Web. 2016-07-01 [PY3_PL15] Larry Hastings (write-up by Jake Edge), Making Python 3 more attractive, Python Language Summit 2015, PyCon 2015, LWN.net, https://lwn.net/Articles/640179, Web. 2016-07-01 [ICLD] IronPython team, Ironclad, https://github.com/IronLanguages/ ironclad, 2015-01-02, Web. 2016-07-01 [IRPY] Jim Hugunin, Dino Viehland, Jeff Hardy, Microsoft, IronPython – the Python programming language for the .NET Framework, http://ironpython.net, 2014-12-06, Web. 2016-07-01 [CPYEXT] PyPy team, PyPy/Python compatibility, http://pypy.org/compat. html, Web. 2016-07-01 [JEP] Mike Johnson/Jep Team, Jep - Java Embedded Python, https: //github.com/mrj0/jep, 2015-09-13, Web. 2016-07-01 [JPY] Brockmann Consult GmbH, jpy, https://github.com/bcdev/jpy, 2015-10-30, Web. 2016-07-01 [JREF] Peter Haggar, IBM Corporation, https://www.ibm.com/ developerworks/library/j-refs, 2002-10-01, Web. 2016-07-01 [JyNI_GSoC] Stefan Richthofer, blog for Google Summer of Code 2015 project “Jython: Add full gc support to JyNI (and support ctypes)”, http://gsoc2015-jyni.blogspot.de/2016/ 01/follow-up-report.html, 2016-01-21, Web. 2016-07-01 48 PROC. OF THE 8th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2015)