Cython Cython Build Software That Combines Python’S Expressivity with the Performance “Cython Has Proven Itself and Control of C (And C++)
Total Page：16
File Type：pdf, Size：1020Kb
Cython Cython Build software that combines Python’s expressivity with the performance “Cython has proven itself and control of C (and C++). It’s possible with Cython, the compiler and in many foundational hybrid programming language used by foundational packages such as NumPy, and prominent in projects including Pandas, h5py, and scikits- Python projects where learn. In this practical guide, you’ll learn how to use Cython to improve performance is critical. Python’s performance—up to 3000x—and to wrap C and C++ libraries in In this book, Kurt Smith Python with ease. walks you through Author Kurt Smith takes you through Cython’s capabilities, with sample everything you need to code and in-depth practice exercises. If you’re just starting with Cython, or want to go deeper, you’ll learn how this language is an essential part of any know to tap into Cython's performance-oriented Python programmer’s arsenal. power for your own projects.” ■ Use Cython’s static typing to speed up Python code —Robert Bradshaw Cython ■ Gain hands-on experience using Cython features to boost your Lead Cython Developer numeric-heavy Python A GUIDE FOR PYTHON PROGRAMMERS ■ Create new types with Cython—and see how fast object- oriented programming in Python can be ■ Effectively organize Cython code into separate modules and packages without sacrificing performance ■ Use Cython to give Pythonic interfaces to C and C++ libraries ■ Optimize code with Cython’s runtime and compile-time profiling tools ■ Use Cython’s prange function to parallelize loops transparently with OpenMP Kurt W. Smith discovered Cython’s power during his graduate research in com- putational plasma physics. Today, he uses Cython extensively in his consulting work at Enthought, training scientists, engineers, and researchers in Python, NumPy, Cython, and parallel and high-performance computing. Smith PROGRAMMING/PYTHON Twitter: @oreillymedia facebook.com/oreilly US $29.99 CAN $34.99 ISBN: 978-1-491-90155-7 Kurt W. Smith Cython Cython Build software that combines Python’s expressivity with the performance “Cython has proven itself and control of C (and C++). It’s possible with Cython, the compiler and in many foundational hybrid programming language used by foundational packages such as NumPy, and prominent in projects including Pandas, h5py, and scikits- Python projects where learn. In this practical guide, you’ll learn how to use Cython to improve performance is critical. Python’s performance—up to 3000x—and to wrap C and C++ libraries in In this book, Kurt Smith Python with ease. walks you through Author Kurt Smith takes you through Cython’s capabilities, with sample everything you need to code and in-depth practice exercises. If you’re just starting with Cython, or want to go deeper, you’ll learn how this language is an essential part of any know to tap into Cython's performance-oriented Python programmer’s arsenal. power for your own projects.” ■ Use Cython’s static typing to speed up Python code —Robert Bradshaw Cython ■ Gain hands-on experience using Cython features to boost your Lead Cython Developer numeric-heavy Python A GUIDE FOR PYTHON PROGRAMMERS ■ Create new types with Cython—and see how fast object- oriented programming in Python can be ■ Effectively organize Cython code into separate modules and packages without sacrificing performance ■ Use Cython to give Pythonic interfaces to C and C++ libraries ■ Optimize code with Cython’s runtime and compile-time profiling tools ■ Use Cython’s prange function to parallelize loops transparently with OpenMP Kurt W. Smith discovered Cython’s power during his graduate research in com- putational plasma physics. Today, he uses Cython extensively in his consulting work at Enthought, training scientists, engineers, and researchers in Python, NumPy, Cython, and parallel and high-performance computing. Smith PROGRAMMING/PYTHON Twitter: @oreillymedia facebook.com/oreilly US $29.99 CAN $34.99 ISBN: 978-1-491-90155-7 Kurt W. Smith Cython Kurt W. Smith Cython by Kurt W. Smith Copyright © 2015 Kurt W. Smith, PhD. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or [email protected]. Editors: Mike Loukides and Meghan Blanchette Indexer: Ellen Troutman Zaig Production Editor: Matthew Hacker Cover Designer: Ellie Volckhausen Copyeditor: Rachel Monaghan Interior Designer: David Futato Proofreader: Rachel Head Illustrator: Rebecca Demarest January 2015: First Edition Revision History for the First Edition: 2015-01-09: First release See http://oreilly.com/catalog/errata.csp?isbn=9781491901557 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Cython, the cover image of a South African python, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the author have used good faith efforts to ensure that the information and instruc‐ tions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intel‐ lectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. ISBN: 978-1-491-90155-7 [LSI] For Ellen, Zélie, Leo, and Hugh—my muses. An inconvenience is only an adventure wrongly considered; an adventure is an inconvenience rightly considered. — GKC A dead thing can go with the stream, but only a living thing can go against it. — GKC Table of Contents Preface. xi 1. Cython Essentials. 1 Comparing Python, C, and Cython 2 Function Call Overhead 5 Looping 6 Math Operations 6 Stack Versus Heap Allocation 6 Tempering Our Enthusiasm 7 Wrapping C Code with Cython 8 Summary 10 2. Compiling and Running Cython Code. 11 The Cython Compilation Pipeline 12 Installing and Testing Our Setup 13 The Standard Way: Using distutils with cythonize 14 Our distutils Script 15 Compiling with distutils on Mac OS X and Linux 15 Compiling with distutils on Windows 16 Using Our Extension Module 17 Interactive Cython with IPython’s %%cython Magic 19 Compiling On-the-Fly with pyximport 21 Controlling pyximport and Managing Dependencies 22 pyximport Example with External Dependencies 23 Rolling Our Own and Compiling by Hand 24 Using Cython with Other Build Systems 26 CMake and Cython 26 SCons and Cython 26 Make and Cython 26 v Compiler Directives 28 Summary 29 3. Cython in Depth. 31 Interpreted Versus Compiled Execution 31 Dynamic Versus Static Typing 32 Static Type Declaration with cdef 34 Automatic Type Inference in Cython 36 C Pointers in Cython 37 Mixing Statically and Dynamically Typed Variables 39 Statically Declaring Variables with a Python Type 41 Static Typing for Speed 43 Reference Counting and Static String Types 45 Cython’s Three Kinds of Functions 46 Python Functions in Cython with the def Keyword 46 C Functions in Cython with the cdef Keyword 49 Combining def and cdef Functions with cpdef 50 Functions and Exception Handling 51 Functions and the embedsignature Compiler Directive 53 Type Coercion and Casting 55 Declaring and Using structs, unions, and enums 56 Type Aliasing with ctypedef 59 Cython for Loops and while Loops 61 Guidelines for Efficient Loops 61 Loop Example 62 The Cython Preprocessor 63 Bridging the Python 2 and Python 3 Divide 64 str, unicode, bytes, and All That 66 Summary 67 4. Cython in Practice: N-Body Simulation. 69 Overview of the N-Body Python Code 69 Converting to Cython 71 Python Data Structures and Organization 72 Converting Data Structures to structs 73 Running the Cythonized Version 75 Summary 76 5. Cython and Extension Types. 79 Comparing Python Classes and Extension Types 79 Extension Types in Cython 80 Type Attributes and Access Control 83 vi | Table of Contents C-Level Initialization and Finalization 85 cdef and cpdef Methods 86 Inheritance and Subclassing 89 Casting and Subclasses 90 Extension Type Objects and None 91 Extension Type Properties in Cython 92 Special Methods Are Even More Special 94 Arithmetic Methods 94 Rich Comparisons 96 Iterator Support 98 Summary 99 6. Organizing Cython Code. 101 Cython Implementation (.pyx) and Declaration (.pxd) Files 102 The cimport Statement 105 Predefined Definition Files 107 Include Files and the include Statement 109 Organizing and Compiling Cython Modules Inside Python Packages 110 Summary 113 7. Wrapping C Libraries with Cython. 115 Declaring External C Code in Cython 115 Cython Does Not Automate Wrapping 117 Declaring External C Functions and typedefs 118 Declaring and Wrapping C structs, unions, and enums 119 Wrapping C Functions 121 Wrapping C structs with Extension Types 122 Constants, Other Modifiers, and Controlling What Cython Generates 125 Error Checking and Raising Exceptions 128 Callbacks 128 Callbacks and Exception Propagation 133 Summary 134 8. Wrapping C++ Libraries with Cython. 135 Simple Example: MT_RNG Class 135 The Wrapper Extension Type 137 Compiling with C++ 138 Using Our Wrapper from Python 139 Overloaded Methods and Functions 140 Operator Overloading 142 C++ Exceptions 144 Stack and Heap Allocation of C++ Instances 145 Table of Contents | vii Working with C++ Class Hierarchies 146 C++ Templates 147 Templated Functions and Cython’s Fused Types 148 Templated Classes 149 Iterators and Nested Classes 150 Included STL Container Class Declarations 151 Memory Management and Smart Pointers 154 Summary 157 9. Cython Profiling Tools. 159 Cython Runtime Profiling 159 Performance Profiling and Annotations 164 Summary 170 10.