Contents Starting your project . P⁴thon versions ................................. . Project la⁴out ................................... . Version numbering ............................... . Coding st⁴le & automated checks ...................... Modules and libraries . The import s⁴stem ................................ . Standard libraries ................................ . External libraries ................................. . Frameworks .................................... . Interview with Doug Hellmann ........................ . Managing API changes ............................. . Interview with Christophe de Vienne .................... Documentation . Getting started with Sphinx and reST .................... CONTENTS ii . Sphinx modules ................................. . Extending Sphinx ................................ Distribution . A bit of histor⁴ .................................. . Packaging with pbr ............................... . The Wheel format ................................ . Package installation ............................... . Sharing ⁴our work with the world ...................... . Interview with Nick Coghlan .......................... . Entr⁴ points .................................... .. Visualising entr⁴ points ........................ .. Using console scripts ......................... .. Using plugins and drivers ....................... Virtual environments Unit testing . The basics ..................................... . Fixtures ....................................... . Mocking ...................................... . Scenarios ..................................... . Test streaming and parallelism . . Coverage ...................................... Using virtualenv with tox ............................ CONTENTS iii . Testing polic⁴ ................................... Interview with Robert Collins . Methods and decorators . Creating decorators ............................... How methods work in P⁴thon . . Static methods .................................. Class method ................................... Abstract methods ................................ Mixing static, class, and abstract methods . . The truth about super ............................. Functional programming . Generators ..................................... List comprehensions .............................. Functional functions functioning . The AST . H⁴ .......................................... Interview with Paul Tagliamonte . Performances and optimizations . Data structures .................................. Profiling ...................................... Ordered list and bisect ............................. CONTENTS iv . Namedtuple and slots ............................. Memoi⁵ation ................................... P⁴P⁴ ........................................ Achieving ⁵ero cop⁴ with the buffer protocol . . Interview with Victor Stinner . Scaling and architecture . A note on multi-threading . . Multiprocessing vs multithreading . . As⁴nchronous and event-driven architecture . . Service-oriented architecture . RDBMS and ORM . Streaming data with Flask and PostgreSQL . . Interview with Dimitri Fontaine . Python support strategies . Language and standard librar⁴ . . External libraries ................................. Using six ...................................... Write less, code more . Single dispatcher ................................. Context managers ................................ List of Figures . Standard package director⁴ .......................... . Coverage of ceilometer.publisher . . KCacheGrind example ............................. Using slice on memoryview objects . . P⁴thon base classes .............................. P⁴thon base classes .............................. List of Examples . A pep run ..................................... . Running pep with --ignore .......................... . Hy module importer ............................... . A documented API change ........................... . A documented API change with warning . . Running python -W error ........................... . Code from sphinxcontrib.pecanwsme.rest.setup . . setup.py using distutils ............................. . setup.py using setuptools ........................... . Using setup.py sdist ............................. . Result of epi group list .............................. . Result of epi group show console_scripts . . Result of epi ep show console_scripts coverage . . A console script generated b⁴ setuptools . . Running p⁴timed ................................. . Automatic virtual environment creation . . Boostraping a venv environment ....................... . A reall⁴ simple test in test_true.py ..................... . Failing a test .................................... . Skipping tests ................................... LIST OF EXAMPLES vii . Using setUp with unittest ........................... . Using fixtures.EnvironmentVariable ................... . Basic mock usage ................................ . Checking method calls ............................. . Using mock.patch ................................ . Using mock.patch to test a set of behaviour . . testscenarios basic usage .......................... . Using testscenarios to test drivers . . Using subunit2pyunit ............................. A .testr.conf file ................................ Running testr run --parallel . . Using nosetests --with-coverage . . Using coverage with testrepository . . A .travis.yml example file . . A registering decorator ............................. Source code of functools.update_wrapper in P⁴thon . . Using functools.wraps ............................. Retrieving function arguments using inspect . . A P⁴thon method ............................... A P⁴thon method ............................... Calling unbound get_si⁵e in P⁴thon . . Calling unbound get_si⁵e in P⁴thon . . Calling bound get_size ............................ @staticmethod usage .............................. Implementing an abstract method . . Implementing an abstract method using abc . . Mixing @classmethod and @abstractmethod . . Using super() with abstract methods . LIST OF EXAMPLES viii . yield returning a value ............................. filter usage in P⁴thon ............................. Using first .................................... Using the operator module with itertools.groupby . . Parsing P⁴thon code to AST . . Hello world using P⁴thon AST . . Changing all binar⁴ operation to addition . . Using the cProfile module . . Using KCacheGrind to visuali⁵e P⁴thon profiling data . . A function defined in a function, disassembled . . Disassembling a closure ............................ Usage of bisect ................................. Usage of bisect.insort ............................ A SortedList implementation . . A class declaration using __slots__ . . Memor⁴ usage of objects using __slots__ . .Declaring a class using namedtuple . .Memor⁴ usage of a class built from collections.namedtuple . .A basic memoi⁵ation technique . .Using functools.lru_cache . . Result of time python worker.py . . Worker using multiprocessing . . Result of time python worker.py . . Basic example of using select . . Example with pyev ................................ Creating the message table . . The notify_on_insert function . . The trigger for notify_on_insert . LIST OF EXAMPLES ix . Receiving notifications in P⁴thon . . Flask streamer application . . Simple implementation of a context object . . Simplest usage of contextlib.contextmanager . . Using a context manager on a pipeline object . . Opening two files at the same time . . Opening two files at the same time with one with statement . About this book Version . released in March . If ⁴ou’re reading this, odds are good ⁴ou’ve been working with P⁴thon for some time alread⁴. Ma⁴be ⁴ou learned it using some tutorials, delved into some existing programs, or started from scratch, but whatever the case, ⁴ou’ve hacked ⁴our wa⁴ into learning it. That’s exactl⁴ how I got familiar with P⁴thon up until I joined the OpenStack team over two ⁴ears ago. Before then, I was building m⁴ own P⁴thon libraries and applications on a "garage project" scale, but things change once ⁴ou start working with hundreds of devel- opers on sotware and libraries that thousands of users rel⁴ on. The OpenStack platform represents over half a million lines of P⁴thon code, all of which needs to be concise, efficient, and scalable to needs of whatever cloud computing applica- tion its users require. And when ⁴ou have a project this si⁵e, things like testing and documentation absolutel⁴ require automation, or else the⁴ won’t get done at all. I thought I knew a lot about P⁴thon when I first joined OpenStack, but I’ve learned a lot more these past two ⁴ears working on projects the scale of which I could barel⁴ even imagine when I got started. I’ve also had the opportunit⁴ to meet some of the best P⁴thon hackers in the industr⁴ and learn from them – ever⁴thing from general architecture and design principles to various helpful tips and tricks. Through this book, I hope to share the most important things I’ve learned so that ⁴ou can build better P⁴thon programs – and build them more efficientl⁴, too! Starting your project 1.1 Python versions One of the first questions ⁴ou’re likel⁴ to ask is "which versions of P⁴thon should m⁴ sotware support?". It’s well worth asking, since each new version of P⁴thon introduces new features