Proceedings of the 8th Python in Science conference Gaël Varoquaux, Stefan van der Walt, Jarrod Millman

To cite this version:

Gaël Varoquaux, Stefan van der Walt, Jarrod Millman. Proceedings of the 8th Python in Science conference. SciPy 2009: 8th Python in Science Conference, Aug 2009, Pasadena, United States. pp.1-78. ￿hal-00502607￿

HAL Id: hal-00502607 https://hal.archives-ouvertes.fr/hal-00502607 Submitted on 15 Jul 2010

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Proceedings of the 8th Python in Science Conference SciPy Conference – Pasadena, CA, August 18-23, 2009.

Editors: Gaël Varoquaux, Stéfan van der Walt, K. Jarrod Millman

Contents

Editorial 2 G. Varoquaux, S. van der Walt, J. Millman Cython tutorial 4 S. Behnel, R. Bradshaw, D. Seljebotn Fast numerical computations with Cython 15 D. Seljebotn High-Performance Code Generation Using CorePy 23 A. Friedley, C. Mueller, A. Lumsdaine Convert-XY: type-safe interchange of C++ and Python containers for NumPy extensions 29 D. Eads, E. Rosten Parallel Kernels: An Architecture for Distributed Parallel Computing 36 P. Kienzle, N. Patel, M. McKerns PaPy: Parallel and distributed data-processing pipelines in Python 41 M. Cieślik, C. Mura PMI - Parallel Method Invocation 48 O. Lenz Sherpa: 1D/2D modeling and fitting in Python 51 B. Refsdal, S. Doe, D. Nguyen, A. Siemiginowska, N. Bonaventura, D. Burke, I. Evans, J. Evans, A. Fruscione, E. Galle, J. Houck, M. Karovska, N. Lee, M. Nowak The FEMhub Project and Classroom Teaching of Numerical Methods 58 P. Solin, O. Certik, S. Regmi Exploring the future of bioinformatics data sharing and mining with Pygr and Worldbase 62 C. Lee, A. Alekseyenko, C. Brown Nitime: time-series analysis for neuroimaging data 68 A. Rokem, M. Trumpis, F. Pérez Multiprocess System for Virtual Instruments in Python 76 B. D’Urso Neutron-scattering data acquisition and experiment automation with Python 81 P. Zolnierczuk, R. Riedel Progress Report: NumPy and SciPy Documentation in 2009 84 J. Harrington, D. Goldsmith

The content of the articles of the Proceedings of the Python in Science Conference is copyrighted and owned by their original authors. For republication or other use of the material published, please contact the copyright owners to obtain permission.

Proceedings of the 8th Python in Science Conference by Gaël Varoquaux, Stéfan van der Walt, K. Jarrod Milllman ISBN: 978-0-557-23212-3 Proceedings of the 8th Python in Science Conference (SciPy 2009)

Organization

Conference chair K. Jarrod Millman UC Berkeley, Helen Wills Neuroscience Institute, USA

Tutorial Co-chairs Dave Peterson Enthought Inc, Austin, USA Fernando Perez UC Berkeley, Helen Wills Neuroscience Institute, USA

Program Co-chairs Gaël Varoquaux INRIA Saclay, France Stéfan van der Walt Stellenbosh University, South Africa

Program Committee Michael Aivazis Center for Advanced Computing Research, California Institute of Technology USA Brian Granger Physics Department, California Polytechnic State University, San Luis Obispo USA Aric Hagberg Theoretical Division, Los Alamos National Laboratory USA Konrad Hinsen Centre de Biophysique Moléculaire, CNRS Orléans France Randall LeVeque Mathematics, University of Washington, Seattle USA Travis Oliphant Enthought Inc. USA Prabhu Ramachandran Department of Aerospace Engineering, IIT Bombay India Raphael Ritz International Neuroinformatics Coordinating Facility Sweden William Stein Mathematics, University of Washington, Seattle USA

Proceeding reviewers Francesc Alted Pytables Spain Philippe Ciuciu CEA, Neurospin France Yann Cointepas CEA, Neurospin France Emmanuelle Gouillart CNRS Saint Gobain France Jonathan Guyer NIST USA Ben Herbst Stellenbosh University, South Africa Paul Kienzle NIST USA Michael McKerns Center for Advanced Computing Research, California Institute of Technology USA Sturla Molden University of Oslo Norway Jean-Baptiste Poline CEA, Neurospin France Dag Sverre Seljebotn University of Oslo Norway Gregor Thalhammer University of Florence Italy

1 Proceedings of the 8th Python in Science Conference (SciPy 2009)

Editorial

Gael Varoquaux ([email protected])– INRIA, Saclay France Stéfan van der Walt ([email protected])– University of Stellenbosch, Stellenbosch South Africa Jarrod Millman ([email protected])– UC Berkeley, Berkeley, CA USA

SciPy 2009 marks our eighth annual Python in Sci- Both keynote addresses served to outline prominent ence conference and the second edition of the confer- themes that were repeated throughout the conference, ence proceedings. The conference and these proceed- as witnessed by the proceedings. These themes in- ings highlight the ongoing focus of the community on clude: the need for software tools that allow scientists providing practical software tools, created to address to focus on their research, while taking advantage of real scientific problems. best-of-class algorithms and utilizing the full power of As in previous years, topics at the conference ranged their computational resources; the need for a high- from the presentation of tools and techniques for scien- level computing environment with an easy—to-write tific work with the Python language, to reports on sci- and read syntax; the usefulness of high-quality soft- entific achievement using Python. Interestingly, sev- ware tools for teaching and education; and the impor- eral people noticed that something important hap- tance of sharing code and data in scientific research. pened in the Scientific Python world during the last The first several articles address high-level approaches year: we are no longer constantly comparing our soft- aimed at improving the performance of numerical code ware with commercial packages, nor justifying the written in Python. While making better use of in- need for Python in Science. Python has now reached creased computation resources, such as parallel pro- the level of adoption where this sort of justification cessors or graphical processing units, many of these is no longer necessary. The exact moment when this approaches also focus on reducing code complexity and shift in focus occurred is difficult to identify, but that verbosity. Again, simpler software allows scientists to it happened was apparent during the conference. focus on the details of their computations, rather than on administrating their computing resources. Recurring scientific themes The remaining articles focus on work done to solve problems in specific research domains, ranging from This year the conference spanned two days, and each numerical methods to biology and astronomy. For the day commenced with a keynote address. The first last several years, using Python to wrap existing li- keynote was delivered by Peter Norvig, the Director braries has been a popular way to provide a scripting of Research at Google; the second by Jonathan Guyer, frontend to computational code written primarily in a a materials scientist in the Thermodynamics and Ki- more low-level language like C or Fortran. However, as netics Group at the National Institute of Standards these proceedings show, Python is increasingly used as and Technology (NIST). the primary language for large scientific applications. Peter Norvig’s talk was titled “What to demand from a Python and its stack of scientific tools appears to be Scientific Computing Language—even if you don’t care well suited for application areas ranging from database about computing or languages”, where he discussed a applications to user interfaces and numerical compu- number of desired characteristics in a scientific com- tation. puting environment. Such a platform should have the ability to share code and data with other researchers Review and selection process easily, provide extremely fast computations and state- of-the-art algorithms to researchers in the field, and be This year we received 30 abstracts from five differ- as easy as possible to use in a time-efficient manner. ent countries. The submissions covered a number of He also stressed the importance of having code that research fields, including bioinformatics, computer vi- read like the mathematical ideas it expressed. sion, nanomaterials, neutron scattering, neuroscience, Jonathan Guyer’s keynote centred around “Modeling applied mathematics, astronomy, and X-ray fluores- of Materials with Python”. He expanded on several cence. Moreover, the articles discussed involve a num- of the above-mentioned characteristics as he discussed ber of computational tools: these include statistical the development of FiPy, a framework for solving par- modeling, data mining, visualization, performance op- tial differential equations based on a finite volume ap- timization, parallel computing, code wrapping, instru- proach. Jonathan explained how FiPy was created to ment control, time series analysis, geographic informa- provide the most advanced numerical techniques to sci- tion science, spatial data analysis, adaptive interpola- entists, so that they could focus on the scientific ques- tion, spectral analysis, symbolic mathematics, finite tions at hand, while having a standard platform for element, and virtual reality. Several abstracts also ad- sharing codes with colleagues. Importantly, FiPy has dressed the role of scientific Python in teaching and become a critical tool in Jonathan’s research group and education. has been adopted by many of their colleagues for both research as well as teaching.

G. Varoquaux, S. van der Walt, J. Millmanin Proc. SciPy 2009, G. Varoquaux, S. van der Walt, J. Millman (Eds), pp. 2–4 2 Proceedings of the 8th Python in Science Conference (SciPy 2009)

Each abstract was reviewed by both the program • Are the code examples (if any) sound, clear, and chairs, as well as two members of the program com- well-written? mittee (PC). The PC consisted of 11 members from five countries, and represented both industry and • Is the paper fit for publication in the SciPy pro- academia. Abstracts were evaluated according to the ceedings? Improvements may be suggested, with or following criteria: without a second review. • Relevance of the contribution, with regard to the topics and goals of the conference. From the 30 original abstracts, 12(40%) have been ac- cepted for publication in these proceedings. • Scientific or technical quality of the work presented. Prior to commencing the conference, we had two days • Originality and soundness. of tutorials with both an introductory and advanced We accepted 23 (76%) submission for oral presentation track. In addition to publishing a selection of the pre- at the conference. At the closure of the conference, we sented work, we also selected one of this year’s tutorial invited the presenters to submit their work for publica- presentations for publication. tion in the conference proceedings. These submissions The proceedings conclude with a short progress report were reviewed by 11 proceeding reviewers from seven on the two-year long NumPy and SciPy documentation countries, according to the following criteria: project. • Does the paper describe a well-formulated scientific or technical achievement? The SciPy Conference has been supported since its • Is the content of the paper accessible to a compu- inception by the Center for Advanced Computing Re- tational scientist with no specific knowledge in the search (CACR) at Caltech and Enthought Inc. In ad- given field? dition, we were delighted to receive funding this year • Are the technical and scientific decisions well- from the Python Software Foundation to cover the motivated? travel, registration, and accommodation expenses of 10 students. Finally, we are very grateful to Leah Jones • Does the paper reference scientific sources and ma- of Enthought and Julie Ponce of the CACR for their terial used? invaluable help in organizing the conference.

3 http://conference.scipy.org/proceedings/SciPy2009/paper_0 Proceedings of the 8th Python in Science Conference (SciPy 2009)

Cython tutorial

Stefan Behnel ([email protected])– Senacor Technologies AG, Germany a Robert W. Bradshaw ([email protected])– University of Washington , USA bcd Dag Sverre Seljebotn ([email protected])– University of Oslo , Norway

aDepartment of Mathematics, University of Washington, Seattle, WA, USA bInstitute of Theoretical Astrophysics, University of Oslo, P.O. Box 1029 Blindern, N-0315 Oslo, Norway cDepartment of Mathematics, University of Oslo, P.O. Box 1053 Blindern, N-0316 Oslo, Norway dCentre of Mathematics for Applications, University of Oslo, P.O. Box 1053 Blindern, N-0316 Oslo, Norway

Cython is a programming language based on Python C types. These allow Cython to assign C semantics to with extra syntax to provide static type declarations. parts of the code, and to translate them into very effi- This takes advantage of the benefits of Python while cient C code. Type declarations can therefore be used allowing one to achieve the speed of C. In this paper for two purposes: for moving code sections from dy- we describe the Cython language and show how it namic Python semantics into static-and-fast C seman- can be used both to write optimized code and to tics, but also for directly manipulating types defined in interface with external C libraries. external libraries. Cython thus merges the two worlds into a very broadly applicable programming language. Cython - an overview Installing Cython [Cython] is a programming language based on Python, with extra syntax allowing for optional static type Many scientific Python distributions, such as the declarations. It aims to become a superset of the Enthought Python Distribution [EPD], Python(x,y) [Python] language which gives it high-level, object- [Pythonxy], and Sage [Sage], bundle Cython and no oriented, functional, and dynamic programming. The setup is needed. Note however that if your distribu- source code gets translated into optimized C/C++ tion ships a version of Cython which is too old you code and compiled as Python extension modules. This can still use the instructions below to update Cython. allows for both very fast program execution and tight Everything in this tutorial should work with Cython integration with external C libraries, while keeping 0.11.2 and newer, unless a footnote says otherwise. up the high programmer productivity for which the Unlike most Python software, Cython requires a C Python language is well known. compiler to be present on the system. The details of The primary Python execution environment is com- getting a C compiler varies according to the system monly referred to as CPython, as it is written in used: C. Other major implementations use Java (Jython • Linux The GNU C Compiler (gcc) is usu- [Jython]), C# (IronPython [IronPython]) and Python ally present, or easily available through the itself (PyPy [PyPy]). Written in C, CPython has been package system. On Ubuntu or Debian, for conducive to wrapping many external libraries that in- instance, the command sudo apt-get install terface through the C language. It has, however, re- build-essential will fetch everything you need. mained non trivial to write the necessary glue code in C, especially for programmers who are more fluent in a • Mac OS X To retrieve gcc, one option is to install high-level language like Python than in a do-it-yourself Apple’s XCode, which can be retrieved from the Mac language like C. OS X’s install DVDs or from http://developer. Originally based on the well-known Pyrex [Pyrex], the apple.com. Cython project has approached this problem by means of a source code compiler that translates Python code • Windows A popular option is to use the open to equivalent C code. This code is executed within the source MinGW (a Windows distribution of gcc). See CPython runtime environment, but at the speed of the appendix for instructions for setting up MinGW compiled C and with the ability to call directly into C manually. EPD and Python(x,y) bundle MinGW, libraries. At the same time, it keeps the original inter- but some of the configuration steps in the appendix face of the Python source code, which makes it directly might still be necessary. Another option is to use Mi- usable from Python code. These two-fold characteris- crosoft’s Visual C. One must then use the same ver- tics enable Cython’s two major use cases: extending sion which the installed Python was compiled with. the CPython interpreter with fast binary modules, and The newest Cython release can always be downloaded interfacing Python code with external C libraries. from http://cython.org. Unpack the tarball or zip While Cython can compile (most) regular Python file, enter the directory, and then run: code, the generated C code usually gains major (and sometime impressive) speed improvements from op- python setup.py install tional static type declarations for both Python and

S. Behnel, R. Bradshaw, D. Seljebotnin Proc. SciPy 2009, G. Varoquaux, S. van der Walt, J. Millman (Eds), pp. 4–15 4 Proceedings of the 8th Python in Science Conference (SciPy 2009)

If you have Python setuptools set up on your system, you should be able to fetch Cython from PyPI and install it using: easy_install cython

For Windows there is also an executable installer avail- able for download.

Building Cython code

Cython code must, unlike Python, be compiled. This happens in two stages: • A .pyx file is compiled by Cython to a .c file, con- taining the code of a Python extension module Figure 1 • The .c file is compiled by a C compiler to a .so The Sage notebook allows transparently edit- ing and compiling Cython code simply by typing file (or .pyd on Windows) which can be import-ed %cython at the top of a cell and evaluate it. Variables directly into a Python session. and functions defined in a Cython cell imported into There are several ways to build Cython code: the running session. • Write a distutils setup.py. • Use pyximport, importing Cython .pyx files as if they were .py files (using distutils to compile and build the background). • Run the cython command-line utility manually to Data types in Cython produce the .c file from the .pyx file, then manually compiling the .c file into a shared object library Cython is a Python compiler. This means that it can or .dll suitable for import from Python. (This is compile normal Python code without changes (with mostly for debugging and experimentation.) a few obvious exceptions of some as-yet unsupported language features). However, for performance-critical • Use the [Sage] notebook which allows Cython code code, it is often helpful to add static type declarations, inline and makes it easy to experiment with Cython as they will allow Cython to step out of the dynamic code without worrying about compilation details nature of the Python code and generate simpler and (see figure 1 below). faster C code - sometimes faster by orders of magni- Currently, distutils is the most common way Cython tude. files are built and distributed. It must be noted, however, that type declarations can make the source code more verbose and thus less read- Building a Cython module using distutils able. It is therefore discouraged to use them with- out good reason, such as where benchmarks prove that Imagine a simple “hello world” script in a file they really make the code substantially faster in a per- hello.pyx: formance critical section. Typically a few types in the def say_hello_to(name): right spots go a long way. Cython can produce an- print(Hello %s!" % name) notated output (see figure 2 below) that can be very The following could be a corresponding setup.py useful in determining where to add types. script: All C types are available for type declarations: integer from distutils.core import setup and floating point types, complex numbers, structs, from distutils.extension import Extension unions and pointer types. Cython can automatically from Cython.Distutils import build_ext and correctly convert between the types on assign- ext_modules = [Extension("hello", ["hello.pyx"])] ment. This also includes Python’s arbitrary size in- teger types, where value overflows on conversion to a setup( C type will raise a Python OverflowError at runtime. name = ’Hello world app’, The generated C code will handle the platform depen- cmdclass = {’build_ext’: build_ext}, ext_modules = ext_modules dent sizes of C types correctly and safely in this case. ) To build, run python setup.py build_ext --inplace. Then simply start a Python session Faster code by adding types and do from hello import say_hello_to and use the imported function as you see fit. Consider the following pure Python code:

5 http://conference.scipy.org/proceedings/SciPy2009/paper_1 Cython tutorial

cdef functions

Python function calls can be expensive, and this is especially true in Cython because one might need to convert to and from Python objects to do the call. In our example above, the argument is assumed to be a C double both inside f() and in the call to it, yet a Python float object must be constructed around the argument in order to pass it. Therefore Cython provides a syntax for declaring a C- style function, the cdef keyword: cdef double f(double) except *: return sin(x**2)

Some form of except-modifier should usually be added, otherwise Cython will not be able to propagate excep- Figure 2 Using the -a switch to the cython command tions raised in the function (or a function it calls). line program (or following a link from the Sage note- Above except * is used which is always safe. An ex- book) results in an HTML report of Cython code cept clause can be left out if the function returns a interleaved with the generated C code. Lines are col- Python object or if it is guaranteed that an exception ored according to the level of “typedness” - white will not be raised within the function call. lines translates to pure C without any Python API A side-effect of cdef is that the function is no longer calls. This report is invaluable when optimizing a function for speed. available from Python-space, as Python wouldn’t know how to call it. Using the cpdef keyword instead of cdef, a Python wrapper is also created, so that the function is available both from Cython (fast, passing typed values directly) and from Python (wrapping val- ues in Python objects). from math import sin Note also that it is no longer possible to change f at runtime. def f(x): return sin(x**2) Speedup: 45 times over pure Python.

def integrate_f(a, b, N): s = 0 Calling external C functions dx = (b-a)/N for i in range(N): It is perfectly OK to do from math import sin to s += f(a+i*dx) return s * dx use Python’s sin() function. However, calling C’s own sin() function is substantially faster, especially Simply compiling this in Cython merely gives a 5% in tight loops. It can be declared and used in Cython speedup. This is better than nothing, but adding some as follows: static types can make a much larger difference. cdef extern from "math.h": With additional type declarations, this might look like: double sin(double) from math import sin cdef double f(double x): def f(double x): return sin(x*x) return sin(x**2) At this point there are no longer any Python wrapper def integrate_f(double a, double b, int N): objects around our values inside of the main for loop, cdef int i and so we get an impressive speedup to 219 times the cdef double s, dx speed of Python. s = 0 dx = (b-a)/N Note that the above code re-declares the function from for i in range(N): math.h to make it available to Cython code. The C s += f(a+i*dx) compiler will see the original declaration in math.h at return s * dx compile time, but Cython does not parse “math.h” and Since the iterator variable i is typed with C semantics, requires a separate definition. the for-loop will be compiled to pure C code. Typing a, When calling C functions, one must take care to link s and dx is important as they are involved in arithmetic in the appropriate libraries. This can be platform- withing the for-loop; typing b and N makes less of a specific; the below example works on Linux and Mac difference, but in this case it is not much extra work OS X: to be consistent and type the entire function. This results in a 24 times speedup over the pure Python version.

c 2009, S. Behnel, R. Bradshaw, D. Seljebotn 6 Proceedings of the 8th Python in Science Conference (SciPy 2009)

from distutils.core import setup from distutils.extension import Extension order to remedy this, without sacrificing speed, we will from Cython.Distutils import build_ext use a cdef class to represent a function on floating point numbers: ext_modules=[ cdef class Function: Extension("demo", cpdef double evaluate(self, double x) except *: ["demo.pyx"], return 0 libraries=["m"]) # Unix-like specific ] Like before, cpdef makes two versions of the method available; one fast for use from Cython and one slower setup( name = "Demos", for use from Python. Then: cmdclass = {"build_ext": build_ext}, cdef class SinOfSquareFunction(Function): ext_modules = ext_modules cpdef double evaluate(self, double x) except *: ) return sin(x**2) If one uses the Sage notebook to compile Cython code, Using this, we can now change our integration exam- one can use a special comment to tell Sage to link in ple: libraries: def integrate(Function f, double a, double b, int N): cdef int i #clib: m cdef double s, dx if f is None: raise ValueError("f cannot be None") Just like the sin() function from the math library, it s = 0 is possible to declare and call into any C library as long dx = (b-a)/N as the module that Cython generates is properly linked for i in range(N): against the shared or static library. A more extensive s += f.evaluate(a+i*dx) return s * dx example of wrapping a C library is given in the section Using C libraries. print(integrate(SinOfSquareFunction(), 0, 1, 10000)) This is almost as fast as the previous code, however Extension types (aka. cdef classes) it is much more flexible as the function to integrate can be changed. It is even possible to pass in a new To support object-oriented programming, Cython sup- function defined in Python-space. Assuming the above ports writing normal Python classes exactly as in code is in the module integrate.pyx, we can do: Python: >>> import integrate class MathFunction(object): >>> class MyPolynomial(integrate.Function): def __init__(self, name, operator): ... def evaluate(self, x): self.name = name ... return 2*x*x + 3*x - 10 self.operator = operator ... >>> integrate.integrate(MyPolynomial(), 0, 1, 10000) def __call__(self, *operands): -7.8335833300000077 return self.operator(*operands) This is about 20 times slower than SinOfSquareFunc- Based on what Python calls a “built-in type”, how- tion, but still about 10 times faster than the origi- ever, Cython supports a second kind of class: exten- nal Python-only integration code. This shows how sion types, sometimes referred to as “cdef classes” due large the speed-ups can easily be when whole loops to the keywords used for their declaration. They are are moved from Python code into a Cython module. somewhat restricted compared to Python classes, but Some notes on our new implementation of evaluate: are generally more memory efficient and faster than generic Python classes. The main difference is that • The fast method dispatch here only works because they use a C struct to store their fields and meth- evaluate was declared in Function. Had evaluate ods instead of a Python dict. This allows them to been introduced in SinOfSquareFunction, the code store arbitrary C types in their fields without requir- would still work, but Cython would have used the ing a Python wrapper for them, and to access fields slower Python method dispatch mechanism instead. and methods directly at the C level without passing • In the same way, had the argument f not been typed, through a Python dictionary lookup. but only been passed as a Python object, the slower Normal Python classes can inherit from cdef classes, Python dispatch would be used. but not the other way around. Cython requires to know the complete inheritance hierarchy in order to • Since the argument is typed, we need to check lay out their C structs, and restricts it to single in- whether it is None. In Python, this would have re- heritance. Normal Python classes, on the other hand, sulted in an AttributeError when the evaluate can inherit from any number of Python classes and ex- method was looked up, but Cython would instead tension types, both in Cython code and pure Python try to access the (incompatible) internal structure code. of None as if it were a Function, leading to a crash So far our integration example has not been very useful or data corruption. as it only integrates a single hard-coded function. In

7 http://conference.scipy.org/proceedings/SciPy2009/paper_1 Cython tutorial

There is a compiler directive nonecheck which turns on In our integration example, we might break it up into checks for this, at the cost of decreased speed. Here’s pxd files like this: how compiler directives are used to dynamically switch 1. Add a cmath.pxd function which defines the C on or off nonecheck: math.h #cython: nonecheck=True functions available from the C header file, # ^^^ Turns on nonecheck globally like sin. Then one would simply do from cmath import sin in integrate.pyx. import cython 2. Add a integrate.pxd so that other modules # Turn off nonecheck locally for the function written in Cython can define fast custom func- @cython.nonecheck(False) tions to integrate. def func(): cdef MyClass obj = None cdef class Function: try: cpdef evaluate(self, double x) # Turn nonecheck on again for a block cpdef integrate(Function f, double a, with cython.nonecheck(True): double b, int N) print obj.myfunc() # Raises exception except AttributeError: Note that if you have a cdef class with attributes, pass the attributes must be declared in the class dec- print obj.myfunc() # Hope for a crash! laration pxd file (if you use one), not the pyx file. Attributes in cdef classes behave differently from at- The compiler will tell you about this. tributes in regular classes: • All attributes must be pre-declared at compile-time Using Cython with NumPy • Attributes are by default only accessible from Cython has support for fast access to NumPy arrays. Cython (typed access) To optimize code using such arrays one must cimport • Properties can be declared to expose dynamic at- the NumPy pxd file (which ships with Cython), and tributes to Python-space declare any arrays as having the ndarray type. The data type and number of dimensions should be fixed cdef class WaveFunction(Function): # Not available in Python-space: at compile-time and passed. For instance: cdef double offset import numpy as np # Available in Python-space: cimport numpy as np cdef public double freq def myfunc(np.ndarray[np.float64_t, ndim=2] A): # Available in Python-space: <...> property period: def __get__(self): myfunc can now only be passed two-dimensional ar- return 1.0 / self. freq rays containing double precision floats, but array in- def __set__(self, value): dexing operation is much, much faster, making it suit- self. freq = 1.0 / value <...> able for numerical loops. Expect speed increases well over 100 times over a pure Python loop; in some cases the speed increase can be as high as 700 times or more. pxd files [Seljebotn09] contains detailed examples and bench- marks. In addition to the .pyx source files, Cython uses .pxd Fast array declarations can currently only be used files which work like C header files - they contain with function local variables and arguments to def- Cython declarations (and sometimes code sections) style functions (not with arguments to cpdef or cdef, which are only meant for sharing C-level declarations and neither with fields in cdef classes or as global vari- with other Cython modules. A pxd file is imported ables). These limitations are considered known defects into a pyx module by using the cimport keyword. and we hope to remove them eventually. In most cir- pxd files have many use-cases: cumstances it is possible to work around these limi- 1. They can be used for sharing external C declara- tations rather easily and without a significant speed tions. penalty, as all NumPy arrays can also be passed as untyped objects. 2. They can contain functions which are well suited for inlining by the C compiler. Such functions Array indexing is only optimized if exactly as many should be marked inline, example: indices are provided as the number of array dimen- sions. Furthermore, all indices must have a native in- cdef inline int int_min(int a, int b): return b if b < a else a teger type. Slices and NumPy “fancy indexing” is not optimized. Examples: 3. When accompanying an equally named pyx file, they provide a Cython interface to the Cython module so that other Cython modules can com- municate with it using a more efficient protocol than the Python one.

c 2009, S. Behnel, R. Bradshaw, D. Seljebotn 8 Proceedings of the 8th Python in Science Conference (SciPy 2009)

def myfunc(np.ndarray[np.float64_t, ndim=1] A): cdef Py_ssize_t i, j Using C libraries for i in range(A.shape[0]): print A[i, 0] # fast Apart from writing fast code, one of the main use cases j = 2*i of Cython is to call external C libraries from Python print A[i, j] # fast code. As seen for the C string decoding functions k = 2*i print A[i, k] # slow, k is not typed above, it is actually trivial to call C functions directly print A[i][j] # slow in the code. The following describes what needs to be print A[i,:] # slow done to use an external C library in Cython code. Py_ssize_t is a signed integer type provided by Imagine you need an efficient way to store integer val- Python which covers the same range of values as is ues in a FIFO queue. Since memory really matters, supported as NumPy array indices. It is the preferred and the values are actually coming from C code, you type to use for loops over arrays. cannot afford to create and store Python int objects Any Cython primitive type (float, complex float and in a list or deque. So you look out for a queue imple- integer types) can be passed as the array data type. mentation in C. For each valid dtype in the numpy module (such as After some web search, you find the C-algorithms li- np.uint8, np.complex128) there is a correspond- brary [CAlg] and decide to use its double ended queue ing Cython compile-time definition in the cimport-ed implementation. To make the handling easier, how- NumPy pxd file with a _t suffix1. Cython structs are ever, you decide to wrap it in a Python extension type also allowed and corresponds to NumPy record arrays. that can encapsulate all memory management. Examples: The C API of the queue implementation, which is de- cdef packed struct Point: fined in the header file libcalg/queue.h, essentially np.float64_t x, y looks like this: typedef struct _Queue Queue; def f(): typedef void *QueueValue; cdef np.ndarray[np.complex128_t, ndim=3] a = \ np.zeros((3,3,3), dtype=np.complex128) Queue *queue_new(void); cdef np.ndarray[Point] b = np.zeros(10, void queue_free(Queue *queue); dtype=np.dtype([(’x’, np.float64), (’y’, np.float64)])) int queue_push_head(Queue *queue, QueueValue data); <...> QueueValue queue_pop_head(Queue *queue); Note that ndim defaults to 1. Also note that NumPy QueueValue queue_peek_head(Queue *queue); record arrays are by default unaligned, meaning data int queue_push_tail(Queue *queue, QueueValue data); is packed as tightly as possible without considering QueueValue queue_pop_tail(Queue *queue); the alignment preferences of the CPU. Such unaligned QueueValue queue_peek_tail(Queue *queue); record arrays corresponds to a Cython packed struct. int queue_is_empty(Queue *queue); If one uses an aligned dtype, by passing align=True to the dtype constructor, one must drop the packed To get started, the first step is to redefine the C API keyword on the struct definition. in a .pxd file, say, cqueue.pxd: Some data types are not yet supported, like boolean cdef extern from "libcalg/queue.h": arrays and string arrays. Also data types describing ctypedef struct Queue: pass data which is not in the native endian will likely never ctypedef void* QueueValue be supported. It is however possible to access such arrays on a lower level by casting the arrays: Queue* new_queue() cdef np.ndarray[np.uint8, cast=True] boolarr = (x < y) void queue_free(Queue* queue) cdef np.ndarray[np.uint32, cast=True] values = \ np.arange(10, dtype=’>i4’) int queue_push_head(Queue* queue, QueueValue data) QueueValue queue_pop_head(Queue* queue) Assuming one is on a little-endian system, the values QueueValue queue_peek_head(Queue* queue) array can still access the raw bit content of the array int queue_push_tail(Queue* queue, QueueValue data) (which must then be reinterpreted to yield valid results QueueValue queue_pop_tail(Queue* queue) on a little-endian system). QueueValue queue_peek_tail(Queue* queue) Finally, note that typed NumPy array variables in bint queue_is_empty(Queue* queue) some respects behave a little differently from untyped arrays. arr.shape is no longer a tuple. arr.shape[0] Note how these declarations are almost identical to is valid but to e.g. print the shape one must do print the header file declarations, so you can often just copy (