Swig Master Class David "Mr. Swig" Beazley http://www.dabeaz.com Presented at PyCon'2008 Copyright (C) 2008, http://www.dabeaz.com 1- 1 An Introduction • Perhaps you've heard about this little Python- extension building tool called "Swig" and wondered what it was all about • Maybe you used it on some simple code and were a little "surprised" that it worked. • Maybe you tried to use it for something more complicated and were a) Overwhelmed, b) Confused, c) Horrified, or d) All of the above. Copyright (C) 2008, http://www.dabeaz.com 1- 2 Swig Essentials • The official website: http://www.swig.org • The short history • First implementation (summer, 1995) • First release (Feb. 1996) • First paper (June 1996, Python Workshop) • And now thousands of users... Copyright (C) 2008, http://www.dabeaz.com 1- 3 About Me • I am the original creator of Swig • I wanted to make it easy for scientists to put scripting interfaces on physics software • Have since worked on a variety of other projects (parsing tools, debuggers, etc.) • I am still involved with Swig, but am coming off of a bit of sabbatical from working on it. Copyright (C) 2008, http://www.dabeaz.com 1- 4 About Swig • Swig has been around for many years and has been actively maintained by a large group of developers • However, its complexity has grown along with its capabilities • Although it is still easy to use on "simple" projects, more advanced tasks can (regrettably) involve a rather steep learning curve. Copyright (C) 2008, http://www.dabeaz.com 1- 5 About this Class • How Swig is put together • How Swig thinks about extension building • How Swig code generation works • How various customization features work Copyright (C) 2008, http://www.dabeaz.com 1- 6 Disclaimers • This is an advanced course • I will assume the following: • You can write Python programs • You can write C/C++ programs • You have made extensions to Python before (by hand or with a tool) Copyright (C) 2008, http://www.dabeaz.com 1- 7 Disclaimers • This class is about important concepts and the "big picture." • This isn't a Swig reference manual or even a detailed tutorial for beginners. • But, the Swig reference manual will (hopefully) make a lot more sense after this class Copyright (C) 2008, http://www.dabeaz.com 1- 8 No Advocacy! • This is not a Swig sales pitch • It's a look inside to see how Swig works • Mostly, I want to demystify parts of it Copyright (C) 2008, http://www.dabeaz.com 1- 9 Format • Course is organized as a discussion with some live examples/demos • You can follow along, but you will need Python, Swig, C/C++ installed on your system • I'm not going to talk about how to configure your environment. • Please stop me to ask questions! Copyright (C) 2008, http://www.dabeaz.com 1- 10 Part I Python Extension Building and Swig Copyright (C) 2008, http://www.dabeaz.com 1- 11 Python Extensions • Python can be extended with C functions /* A simple C function */ double square(double x) { return x*x; } • To do this, you have to write a wrapper PyObject *py_square(PyObject *self, PyObject *args) { double x, result; if (!PyArg_ParseTuple(self,"d",&x)) { return NULL; } result = square(x); return Py_BuildValue("d",result); } Copyright (C) 2008, http://www.dabeaz.com 1- 12 Wrapper Functions • The wrapper serves as glue Python Wrapper C/C++ • It converts values from Python to a low-level representation that C can work with • It converts results from C back into Python Copyright (C) 2008, http://www.dabeaz.com 1- 13 Extension Modules • An extension module is just a collection of wrapper functions • Additional initialization code sets it up /* Method table for extension module */ static PyMethodDef extmethods[] = { {"square", py_square, METH_VARARGS}, {NULL,NULL} } /* initialization function */ void initext() { Py_InitModule("ext",extmethods); } Copyright (C) 2008, http://www.dabeaz.com 1- 14 Packaging of Extensions • Extension modules usually compiled into shared libraries or DLLs (ext.so, ext.pyd, etc.) • The import statement knows to look for such files (along with .py, .pyc, and .pyo files) >>> import ext >>> ext.square(4) 16.0 >>> • There are many details related to the compilation of these modules (but, that's an entirely different tutorial) Copyright (C) 2008, http://www.dabeaz.com 1- 15 The Problem • Writing extension code by hand is annoying • Extremely tedious and error prone • Difficult to maintain • Not at all obvious when you start getting into gnarly C/C++ code (structs, classes, arrays, pointers, templates, etc.) Copyright (C) 2008, http://www.dabeaz.com 1- 16 Extension Tools • Python has a large number of tools that aim to "simplify" the extension building process • Boost.Python • ctypes • SIP • pyfort • Pyrex • Swig • Apologies to anyone I left out Copyright (C) 2008, http://www.dabeaz.com 1- 17 Swig • Swig generates wrappers from C++ headers • Basic idea : You just list everything you want in your extension module using normal C-style declarations • Swig parses those declarations and creates an output file of C/C++ code which you compile to make your module Copyright (C) 2008, http://www.dabeaz.com 1- 18 Sample Swig Interface • Here is a sample Swig specification: %module sample %{ #include "myheader.h" #include "otherheader.h" %} #define PI 3.14159; int foo(int x, int y); double bar(const char *s); struct Spam { int a, b; }; Copyright (C) 2008, http://www.dabeaz.com 1- 19 Sample Swig Interface • Here is a sample Swig specification: %module sample %{ Preamble. #include "myheader.h" #include "otherheader.h" Gives the module name and %} provides declarations needed to get the code to compile #define PI 3.14159; int foo(int x, int y); (usually header files). double bar(const char *s); struct Spam { int a, b; }; Copyright (C) 2008, http://www.dabeaz.com 1- 20 Sample Swig Interface • Here is a sample Swig specification: %module sample %{ #include "myheader.h" #include "otherheader.h" %} #define PI 3.14159; int foo(int x, int y); Declarations. double bar(const char *s); List everything that you struct Spam { want in the extension int a, b; }; module. Copyright (C) 2008, http://www.dabeaz.com 1- 21 Running Swig • Swig is a command-line tool shell % swig -python sample.i shell % • Unless there are errors, it is silent • Invocation of Swig may be hidden away. • For instance, distutils/setuptools runs Swig automatically if you list a .i file as a source. Copyright (C) 2008, http://www.dabeaz.com 1- 22 Swig Output • As output, Swig produces two files shell % ls sample.i sample_wrap.c sample.py shell % • The _wrap.c file is C code that must be compiled in a shared library • The .py file is Python support code that serves as a front-end to the low-level C module • Users import the .py file Copyright (C) 2008, http://www.dabeaz.com 1- 23 Building Swig Extensions • Swig extension modules always come in pairs sample.i swig sample_wrap.c cc _sample.pyd sample.py These two files are the "module" Copyright (C) 2008, http://www.dabeaz.com 1- 24 Dual-Module Architecture • Swig uses a dual-module architecture where some code is in C and other code is in Python user imports sample.py imports _sample.pyd • This same approach is used by Python itself (socket.py, _socket.pyd, thread.pyd, threading.py) Copyright (C) 2008, http://www.dabeaz.com 1- 25 Using a Swig Module • Usually no big surprises >>> import sample >>> sample.foo(73,37) 42 >>> sample.PI 3.1415926 >>> x = sample.bar("123.45") >>> s = sample.Spam() >>> s.a = 1 >>> s.b = 2 >>> print s.a + s.b 3 >>> • Everything in the declaration list is available and "works" as you would expect Copyright (C) 2008, http://www.dabeaz.com 1- 26 General Philosophy • The main goal of Swig is to make a “natural” interface to C/C++ code %module sample >>> import sample class Spam { >>> s = sample.Spam() public: >>> s.a = 42 int a,b; >>> x = s.bar(37) int bar(int); >>> sample.Spam.foo() static foo(void); >>> }; • A very large subset of C/C++ is supported • Use in Python is the same as use in C++ Copyright (C) 2008, http://www.dabeaz.com 1- 27 Swig-generated Code • Swig generates the same kind of code that you would normally write by hand • It creates wrapper functions • It creates a module initialization function • It packages everything up into a file that you can compile into a extension module Copyright (C) 2008, http://www.dabeaz.com 1- 28 The Horror, The Horror • When people first come to Swig, they might look at the output files and have their head explode. • That code is not meant to be read. • However, there are a number of critical things going on in the output... Copyright (C) 2008, http://www.dabeaz.com 1- 29 Compiler Dependencies • Output includes a lot of compiler switches, macros, and other definitions for portability #ifndef SWIGEXPORT # if defined(_WIN32) || defined(__WIN32__) || defined(__CYGWIN__) # if defined(STATIC_LINKED) # define SWIGEXPORT # else # define SWIGEXPORT __declspec(dllexport) # endif # else # if defined(__GNUC__) && defined(GCC_HASCLASSVISIBILITY) # define SWIGEXPORT __attribute__ ((visibility("default"))) # else # define SWIGEXPORT # endif # endif #endif Copyright (C) 2008, http://www.dabeaz.com 1- 30 Runtime Support • The wrapper code also includes a runtime library of about 3000 lines of code • Library functions, macros, etc. • This is needed to deal with more complex aspects of extension modules (especially C++) • A critical part of how modules are packaged Copyright (C) 2008, http://www.dabeaz.com 1- 31 Self-Containment • Many people don't realize that the output of Swig is identical on all platforms • The wrapper code has no third-party dependencies and does not rely on any part of a Swig installation (headers or libraries) • Code generated by Swig can be distributed independently from Swig • End users don't need Swig installed Copyright (C) 2008, http://www.dabeaz.com 1- 32 Part 2 Extension Building and Type Systems Copyright (C) 2008, http://www.dabeaz.com 1- 33 The Extension Problem • The problem of building extension modules is not new---people have been doing this from the beginning days of Python.