Distutils? Setuptools? easy_install??? A brief overview of the Python packaging ecosystem Some terms... ● Package (horribly overloaded) ● A Python package, in the sense of a directory that's also a module containing other modules ● An archive file used by a package management system to install a piece of software ● Distribution ● A collection of Python packages/modules as well as other source/non-source files distributed together as a single product, along with a setup.py (commonly referred to as a “package”, but that name is already taken in Python) More terms... ● Project/Product ● My preferred terms for a single software product with its own name, distributed by itself (though not necessarily without dependencies) (i.e. PyFITS, pysynphot) ● Packaging ● The process of collecting the pieces of a product into a distributable form that other users can download/build/install And more terms... ● Release ● A specific version of a product, as the result of a development cycle (i.e. PyFITS 3.0) ● May have multiple source and/or binary distributions The Python packaging ecosystem distutils ● The original Python packaging/distribution system ● Included in the stdlib ● Familiar—based on setup.py script ● Supports creating source distribution (tar.gz, and other formats) as well as simple Windows .msi installers ● (slightly messy internals, but not too bad) setuptools ● Third-party packaging system built on top of distutils ● Looks mostly like distutils, but setup.py uses: from setuptools import setup instead of: from distutils.core import setup ● Includes easy_install for automatic installation of distributions from the Cheeseshop (now known formally as PyPI) setuptools (cont.) ● Adds support for 'eggs'—a type of binary distribution of a Python release meant for easy drop-in installation (never quite lived up to their promise, though are useful in some areas) ● ./setup.py install and easy_install create eggs out of all Python distributions (even ones that don't use setuptools by default) ● Widely used due mainly to easy_install ● (confusing, fairly messy internals; often buggy) distribute distribute (my take) distribute ● A fork of setuptools—internally looks exactly like setuptools with all the warts ● Still installs a Python package called 'setuptools'; if setuptools is already installed distribute replaces it ● There are no Python packages called 'distribute' ● Mostly created out of political necessity—is somewhat better maintained ● Main advantage is built-in support for 2to3 ● Stronger community support at this point distribute (cont.) ● Based on a direct fork of setuptools, so internals are still a mess ● The plan for distribute was originally to start cleaning it up and improving the quality, but attention turned instead to distutils2 ● More actively maintained than setuptools, but some things slip through the cracks (especially now that setuptools is occasionally being updated again) pip ● Not a packaging system, just a downloader/installer for Python distributions ● Replacement for easy_install, has some advantages: ● Ensures all dependencies can be installed before installing a package; can roll back on failure ● Built in uninstall support ● Doesn't install eggs ● Doesn't automatically update distributions to latest versions—easier reproducibility pip (cont.) ● More advantages: ● Somewhat cleaner/less noisy output than easy_install ● Support for installation from more VCSs ● Cleaner, relatively more understandable code ● Again, just an installer—projects must still be distributed with distutils/setuptools/distribute ● May no longer be necessary when distutils2/packaging comes out distutils2 ● The end-all-be-all replacement for distutils in the standard library ● A complete rewrite—fairly clean design based on many lessons learned ● Had to be a new library rather than updating distutils due to the long history of distutils and the many existing products that rely on its internals ● Primary a tool for installing/uninstalling Python distributions ● Also provides a framework for building more sophisticated package managers and tools distutils2 (cont.) ● Much of the effort that was originally planned for distribute shifted to distutils2 ● Does away with setup.py scripts ● A distribution is instead described mostly in the setup.cfg file ● Though it's also possible to provide hook scripts for more complicated builds/installs ● Includes the pysetup command which can build/install from source or from a package index (PyPI); replacement for easy_install/pip packaging ● packaging = distutils2 ● When moved into the stdlib distutils2 was renamed 'packaging' ● Confusing name, but somewhat more clear about its purpose than 'distutils2' ● Planned for inclusion in Python 3.3 ● A standalone version of distutils2 will remain available for Python >= 2.4 ● (Will have to `try: import packaging except ImportError: import distutils2 as packaging`, but oh well) Summary ● Python packaging/distribution frameworks ● distutils ● setuptools ● distribute ● distutils2/packaging ● Python distribution installers ● setup.py install [distutils, setuptools, distribute] ● easy_install [setuptools, distribute] ● pip ● pysetup [distutils2/packaging] .