Python Packaging Guide Release PyPA January 23, 2014 Contents 1 Introduction 3 1.1 What’s covered in the guide.......................................3 1.2 Where to get packaging support.....................................3 1.3 How to help packaging..........................................3 1.4 About the guide.............................................4 1.5 About ‘Page Status’...........................................4 2 Installation & Packaging Quickstart5 2.1 Install the Tools.............................................5 2.2 Create a Virtual Environment......................................5 2.3 Install Python Packages.........................................5 2.4 Cache Wheels..............................................6 2.5 Create your own Project.........................................6 2.6 Build & Upload your Project to PyPI..................................7 3 Tool Recommendations 9 3.1 Installation Tool Recommendations...................................9 3.2 Packaging Tool Recommendations................................... 10 4 The Future of Python Packaging 11 4.1 Goals................................................... 11 4.2 Presentations & Articles......................................... 11 4.3 Completed work............................................. 12 4.4 Work in Progress............................................. 12 4.5 Future Work............................................... 13 5 Advanced Topics 15 5.1 Technical Details............................................. 15 5.2 Platform Integration........................................... 17 5.3 Binary Extensions............................................ 20 6 Project Summaries 25 6.1 pip.................................................... 25 6.2 virtualenv................................................. 25 6.3 setuptools................................................. 25 6.4 wheel................................................... 25 6.5 distlib................................................... 26 6.6 Warehouse................................................ 26 6.7 buildout.................................................. 26 i 6.8 bento................................................... 26 6.9 conda................................................... 26 6.10 Hashdist................................................. 26 7 PEP Summaries 29 7.1 PEP376 Database of Installed Python Distributions........................... 29 7.2 PEP425 Compatibility Tags for Built Distributions........................... 29 7.3 PEP427 The Wheel Binary Package Format 1.0............................. 30 7.4 PEP438 Transitioning to release-file hosting on PyPI.......................... 30 7.5 PEP453 Explicit bootstrapping of pip in Python installations...................... 30 7.6 PEP426 Metadata for Python Software Packages 2.0.......................... 30 7.7 PEP440 Version Identification and Dependency Specification..................... 31 7.8 PEP458 Surviving a Compromise of PyPI................................ 31 8 A Packaging Timeline 33 9 Glossary 35 ii Python Packaging Guide, Release Page Status Incomplete 1 Last Reviewed 2014-01-22 The “Python Packaging User Guide” (PUG) aims to be the authoritative resource on how to package and install distributions in Python using current tools, but also on the efforts to improve Python packaging. The guide is part of a larger effort to improve all of the packaging and installation docs, including pip, setuptools, virtualenv, wheel, and docs.python.org. Ultimately, users need more than a guide to feel confident about the current tools. They need complete, accurate and inter-consistent documentation across all the projects. The guide is maintained here by the Python Packaging Authority (PyPA). It was forked from the “Hitchhikers Guide to Packaging” in March 2013. 1 The main index page will only be marked “Complete”, when all pages are marked “Complete”, which they are not. Contents 1 Python Packaging Guide, Release 2 Contents CHAPTER 1 Introduction Page Status Complete Last Reviewed 2014-01-22 1.1 What’s covered in the guide • An Installation & Packaging Quickstart for getting going fast. • Recommendations & Justifications for what tools you should be using now. • The Future of Python Packaging. • Advanced Topics in Packaging and Installation. • Summaries of the most relevant packaging projects. • Summaries of the most relevant packaging PEPs. • A Packaging Timeline. •A Glossary of packaging and installation terms. 1.2 Where to get packaging support • For support related to a specific project, see the mailing list and IRC links on the Projects page. • For something more general, or when you just not sure, use the distutils-sig list. 1.3 How to help packaging • Get involved with one of mainstream packaging projects. • Help us catalog and discuss the current problems in packaging and installation. See the The issue tracker for the problems in packaging maintained by PyPA. • Discuss PEPs and long terms plans at distutils-sig. 3 Python Packaging Guide, Release 1.4 About the guide The guide is maintained here on Github by the Python Packaging Authority (PyPA). It was forked from the “Hitchhikers Guide to Packaging” in March 2013. Please Log Issues and Pull Requests to help make the guide better. 1.5 About ‘Page Status’ Each page, even this one, will state at the top whether it’s “Complete” or “Incomplete”. Admittedly, a binary distinction is not very precise, but most pages will additionally offer a footnote to better explain what’s not complete or wrong. Also, each page will give a “Last Reviewed” date, wich will only be updated by a PyPA member when a meaningful review is done to determine the status. Why do this? See here. 4 Chapter 1. Introduction CHAPTER 2 Installation & Packaging Quickstart Page Status InComplete Last Reviewed 2014-01-22 2.1 Install the Tools 1. Securely Download get-pip.py 1 2. Run python get-pip.py (this will install pip and setuptools) 23 3. Run pip install virtualenv 2.2 Create a Virtual Environment virtualenv myVE source myVE/bin/activate Or, if you want to install packages globally, don’t do this. 2.3 Install Python Packages Install SomePackage and it’s dependencies from PyPI using Requirement Specifiers pip install SomePackage # latest version pip install SomePackage==1.0.4 # specific version pip install ’SomePackage>=1.0.4’ # minimum version Install a list of requirements specified in a Requirements File. pip install -r requirements.txt Upgrade an already installed SomePackage to the latest from PyPI. 1 “Secure” in this context means using a modern browser or a tool like curl that verifies SSL certificates when downloading from https URLs. 2 Depending on your platform, this may require root or Administrator access. 3 On Linux and OSX, these tools will usually be available for the system python from a system package manager (e.g. yum or apt-get for linux, or homebrew for OSX). Unfortunately, there is often delay in getting the latest version this way, so in most cases, you’ll want to use the instructions. 5 Python Packaging Guide, Release pip install --upgrade SomePackage Install a project from VCS in “editable” mode. For a full breakdown of the syntax, see pip’s section on VCS Support. pip install -e git+https://git.repo/some_pkg.git#egg=SomePackage # from git pip install -e hg+https://hg.repo/some_pkg.git#egg=SomePackage # from mercurial pip install -e svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage # from svn pip install -e git+https://git.repo/some_pkg.git@feature#egg=SomePackage # from a branch Install a particular source archive file. pip install ./downloads/SomePackage-1.0.4.tar.gz pip install http://my.package.repo/SomePackage-1.0.4.zip Install from an alternate index pip install --index-url http://my.package.repo/simple/ SomePackage Search an additional index during install, in addition to PyPI pip install --extra-index-url http://my.package.repo/simple SomePackage Install from a local directory containing archives (and don’t check PyPI) pip install --no-index --find-links=file:///local/dir/ SomePackage pip install --no-index --find-links=/local/dir/ SomePackage pip install --no-index --find-links=relative/dir/ SomePackage Find pre-release and development versions, in addition to stable versions. By default, pip only finds stable versions. pip install --pre SomePackage For more on installation, see the pip docs. 2.4 Cache Wheels FIXME, cover ’pip wheel’ 2.5 Create your own Project See the PyPA sample project. You can can copy and edit from that to get your project going. To install your project in “develop” or “editable” mode (i.e. to have your project installed, but still editable for development) cd myproject python setup.py develop # the setuptools way pip install -e . # the pip way For more information on creating projects, see: • Setuptools Developer Guide • Open Sourcing a Python Project the Right Way 6 Chapter 2. Installation & Packaging Quickstart Python Packaging Guide, Release 2.6 Build & Upload your Project to PyPI Build a source distribution python setup.py sdist Build a wheel (for advice on when, see Should you upload wheels to PyPI?) pip install wheel python setup.py bdist_wheel Upload your distributions with twine pip install twine twine upload dist/* 2.6. Build & Upload your Project to PyPI 7 Python Packaging Guide, Release 8 Chapter 2. Installation & Packaging Quickstart CHAPTER 3 Tool Recommendations Page Status Incomplete Last Reviewed 2014-01-22 If you’re familiar with Python packaging and installation, and just want to know what tools are currently recommended, then here it