Package software for any distribution with upt
Cyril Roelandt
Package software for any distribution with upt 1/28 Introduction
Cyril Roelandt OpenStack (Red Hat) GNU Guix (Do not miss Ludovic’s talk at 4:30PM!)
Package software for any distribution with upt 2/28 Outline
1 Packaging
2 Automation
3 The Universal Packaging Tool
4 fpm
5 Conclusion
Package software for any distribution with upt 3/28 Outline
1 Packaging
2 Automation
3 The Universal Packaging Tool
4 fpm
5 Conclusion
Package software for any distribution with upt 4/28 Debugging Send/maintain patches
What is packaging?
Turn an “upstream” package into a “downstream” package
Package software for any distribution with upt 5/28 Send/maintain patches
What is packaging?
Turn an “upstream” package into a “downstream” package Debugging
Package software for any distribution with upt 5/28 What is packaging?
Turn an “upstream” package into a “downstream” package Debugging Send/maintain patches
Package software for any distribution with upt 5/28 Dependency hell Debugging
Package layout Metadata
Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated:
Some tasks can be automated:
Is packaging hard?
Many sources (PyPI, RubyGems. . . )
Package software for any distribution with upt 6/28 Dependency hell Debugging
Package layout Metadata
Cannot be fully automated:
Some tasks can be automated:
Is packaging hard?
Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . )
Package software for any distribution with upt 6/28 Package layout Metadata
Dependency hell Debugging Some tasks can be automated:
Is packaging hard?
Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated:
Package software for any distribution with upt 6/28 Package layout Metadata
Debugging Some tasks can be automated:
Is packaging hard?
Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell
Package software for any distribution with upt 6/28 Package layout Metadata
Some tasks can be automated:
Is packaging hard?
Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Debugging
Package software for any distribution with upt 6/28 Package layout Metadata
Is packaging hard?
Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Debugging Some tasks can be automated:
Package software for any distribution with upt 6/28 Metadata
Is packaging hard?
Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Debugging Some tasks can be automated: Package layout
Package software for any distribution with upt 6/28 Is packaging hard?
Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Debugging Some tasks can be automated: Package layout Metadata
Package software for any distribution with upt 6/28 Outline
1 Packaging
2 Automation
3 The Universal Packaging Tool
4 fpm
5 Conclusion
Package software for any distribution with upt 7/28 Existing tools
Debian Fedora Guix FreeBSD OpenBSD CPAN dh-make-perl cpan2rpm guix import ? PortGen NPM npm2deb npm2rpm N/A ? ? PyPI pypi2deb pyp2rpm guix import pytoport PortGen Ruby gem2deb gem2rpm guix import ? PortGen
Package software for any distribution with upt 8/28 Different behaviours Code duplication (read upstream archive) Code duplication (write downstream package)
Issues
One CLI per tool
Package software for any distribution with upt 9/28 Code duplication (read upstream archive) Code duplication (write downstream package)
Issues
One CLI per tool Different behaviours
Package software for any distribution with upt 9/28 Code duplication (write downstream package)
Issues
One CLI per tool Different behaviours Code duplication (read upstream archive)
Package software for any distribution with upt 9/28 Issues
One CLI per tool Different behaviours Code duplication (read upstream archive) Code duplication (write downstream package)
Package software for any distribution with upt 9/28 License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996
For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .
Classifiers:
“license” field (setup.{cfg,py})
Example: licenses (PyPI)
SPDX identifiers?
Package software for any distribution with upt 10/28 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .
License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py})
Example: licenses (PyPI)
SPDX identifiers? Classifiers:
Package software for any distribution with upt 10/28 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .
BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py})
Example: licenses (PyPI)
SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License
Package software for any distribution with upt 10/28 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .
https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py})
Example: licenses (PyPI)
SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ?
Package software for any distribution with upt 10/28 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .
“license” field (setup.{cfg,py})
Example: licenses (PyPI)
SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996
Package software for any distribution with upt 10/28 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .
Example: licenses (PyPI)
SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py})
Package software for any distribution with upt 10/28 ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .
Example: licenses (PyPI)
SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py}) For values not available in the standard classifiers
Package software for any distribution with upt 10/28 Example: licenses (PyPI)
SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py}) For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . .
Package software for any distribution with upt 10/28 Example: dependencies (PyPI)
pipenv $ curl -s https://pypi.org/pypi/pipenv/json \ | jq .info.requires_dist [ "ordereddict; python_version <\"2.7\"", "requests[security]; python_version <\"2.7\"", "virtualenv", "virtualenv-clone (>=0.2.5)", "setuptools (>=36.2.1)", "certifi", "pip (>=9.0.1)" ]
Package software for any distribution with upt 11/28 $ wget https://.../botocore-1.12.16-py2.py3-none-any.whl $ unzip botocore-1.12.16-py2.py3-none-any.whl $ cd botocore-1.12.16.dist-info/ $ jq .run_requires metadata.json [ { "requires":[ "jmespath>=0.7.1,<1.0.0", "docutils>=0.10", "urllib3>=1.20,<1.24" ] }, ... ]
Example: dependencies (PyPI) botocore $ curl -s https://pypi.org/pypi/botocore/json \ | jq .info.requires_dist null
Package software for any distribution with upt 12/28 Example: dependencies (PyPI) botocore $ curl -s https://pypi.org/pypi/botocore/json \ | jq .info.requires_dist null $ wget https://.../botocore-1.12.16-py2.py3-none-any.whl $ unzip botocore-1.12.16-py2.py3-none-any.whl $ cd botocore-1.12.16.dist-info/ $ jq .run_requires metadata.json [ { "requires":[ "jmespath>=0.7.1,<1.0.0", "docutils>=0.10", "urllib3>=1.20,<1.24" ] }, ... Package software] for any distribution with upt 12/28 $ pip download --only-binary :all: MarkupSafe Collecting MarkupSafe Could not find a version that satisfies the \ requirement MarkupSafe (from versions: ) No matching distribution found for MarkupSafe $ cat setup.py ... extras_require={ 'dev': ... 'docs': ... }
Example: dependencies (PyPI)
MarkupSafe $ curl -s https://pypi.org/pypi/MarkupSafe/json \ | jq .info.requires_dist null
Package software for any distribution with upt 13/28 $ cat setup.py ... extras_require={ 'dev': ... 'docs': ... }
Example: dependencies (PyPI)
MarkupSafe $ curl -s https://pypi.org/pypi/MarkupSafe/json \ | jq .info.requires_dist null $ pip download --only-binary :all: MarkupSafe Collecting MarkupSafe Could not find a version that satisfies the \ requirement MarkupSafe (from versions: ) No matching distribution found for MarkupSafe
Package software for any distribution with upt 13/28 Example: dependencies (PyPI)
MarkupSafe $ curl -s https://pypi.org/pypi/MarkupSafe/json \ | jq .info.requires_dist null $ pip download --only-binary :all: MarkupSafe Collecting MarkupSafe Could not find a version that satisfies the \ requirement MarkupSafe (from versions: ) No matching distribution found for MarkupSafe $ cat setup.py ... extras_require={ 'dev': ... 'docs': ... }
Package software for any distribution with upt 13/28 Example: dependencies (PyPI)
Read the JSON file provided by PyPI Read the wheel Read the source code Install the package, run “pip list” What about test dependencies? There might be even more ideas!
Package software for any distribution with upt 14/28 Unified behaviour Modular Simple
What we really need
Unified CLI
Package software for any distribution with upt 15/28 Modular Simple
What we really need
Unified CLI Unified behaviour
Package software for any distribution with upt 15/28 Simple
What we really need
Unified CLI Unified behaviour Modular
Package software for any distribution with upt 15/28 What we really need
Unified CLI Unified behaviour Modular Simple
Package software for any distribution with upt 15/28 Outline
1 Packaging
2 Automation
3 The Universal Packaging Tool
4 fpm
5 Conclusion
Package software for any distribution with upt 16/28 Architecture
Upstream | upt | upt | upt | Downstream Repos | Frontends | | Backends | Packages ------+------+ | +------+ | +-----+ | +------+ | +------+ | ||| |||||| ||| | | PyPI |-|->| upt-pypi |-|->| |-|->| upt-guix |-|->| Guix pkg | | ||| |||||| ||| | +------+ | +------+ | | | | +------+ | +------+ | | | upt | | | +------+ | +------+ | | | | +------+ | +------+ | ||| |||||| ||| | | CPAN |-|->| upt-cpan |-|->| |-|->| upt-openbsd |-|->| Makefile | | ||| |||||| ||| | +------+ | +------+ | +-----+ | +------+ | +------+ | | | |
Package software for any distribution with upt 17/28 Available frontends $ upt list-frontends cpan pypi rubygems
Available backends $ upt list-backends fedora freebsd guix nix openbsd
Usage Install (with modules) $ pip install upt[frontends,backends]
Package software for any distribution with upt 18/28 Available backends $ upt list-backends fedora freebsd guix nix openbsd
Usage Install (with modules) $ pip install upt[frontends,backends]
Available frontends $ upt list-frontends cpan pypi rubygems
Package software for any distribution with upt 18/28 Usage Install (with modules) $ pip install upt[frontends,backends]
Available frontends $ upt list-frontends cpan pypi rubygems
Available backends $ upt list-backends fedora freebsd guix nix openbsd
Package software for any distribution with upt 18/28 Usage
Package upt for OpenBSD $ upt package -f pypi -b openbsd -o /tmp/py-upt upt [INFO ] [Backend] Creating the directory \ structure in /tmp/py-upt [INFO ] [Backend] Creating the Makefile [INFO ] [Backend] Creating pkg/DESCR [INFO ] [Backend] You still need to create \ /tmp/py-upt/distinfo [INFO ] [Backend] You still need to create \ /tmp/py-upt/pkg/PLIST
Package software for any distribution with upt 19/28 Usage $ less /tmp/py-upt/Makefile # $OpenBSD$ COMMENT= package software from any package manager to any distribution MODPY_EGG_VERSION= 0.4.1 DISTNAME= upt-${MODPY_EGG_VERSION} PKGNAME= py-upt-${MODPY_EGG_VERSION} CATEGORIES= XXX HOMEPAGE= https://framagit.org/upt/upt MAINTAINER= XXX
Package software for any distribution with upt 20/28 Usage $ upt package -f pypi -b nix upt { stdenv, buildPythonPackage, fetchPypi , spdx-lookup }: buildPythonPackage rec { name ="${pname}-${version}" ; version = "0.4.1"; pname = "upt"; src = fetchPypi { inherit pname version; sha256 = "TODO"; }; propagatedBuildInputs = [ spdx-lookup ]; meta = with stdenv.lib;{ description = ... homepage = https://framagit.org/upt/upt; license = [ licenses.bsd3 ]; } Package software for any distribution with upt 21/28 } Easier for “small” projects More independence
Modularity
One module (backend or frontend) -> one Git repo
Package software for any distribution with upt 22/28 More independence
Modularity
One module (backend or frontend) -> one Git repo Easier for “small” projects
Package software for any distribution with upt 22/28 Modularity
One module (backend or frontend) -> one Git repo Easier for “small” projects More independence
Package software for any distribution with upt 22/28 Add a backend or frontend $ cookiecutter https://framagit.org/upt/cookiecutter-upt project_name []: upt-archlinux project_slug [upt_archlinux]: author []: John Doe email []: [email protected] short_description []: Arch Linux backend for upt. url [https://framagit.org/upt/upt-archlinux]: Select license: 1 - BSD-3 2 - other Choose from 1, 2 (1, 2) [1]: 1 Select kind: 1 - backend 2 - frontend Choose from 1, 2 (1, 2) [1]: 1 Initialized empty Git repository in /tmp/upt-archlinux/.git/
Package software for any distribution with upt 23/28 Outline
1 Packaging
2 Automation
3 The Universal Packaging Tool
4 fpm
5 Conclusion
Package software for any distribution with upt 24/28 Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager Not the best tool for package maintainers?
A different approcah
Ruby
Package software for any distribution with upt 25/28 Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager Not the best tool for package maintainers?
A different approcah
Ruby Modules are distributed along with the core
Package software for any distribution with upt 25/28 Useful for users who need to use their distro’s package manager Not the best tool for package maintainers?
A different approcah
Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder)
Package software for any distribution with upt 25/28 Not the best tool for package maintainers?
A different approcah
Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager
Package software for any distribution with upt 25/28 A different approcah
Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager Not the best tool for package maintainers?
Package software for any distribution with upt 25/28 Outline
1 Packaging
2 Automation
3 The Universal Packaging Tool
4 fpm
5 Conclusion
Package software for any distribution with upt 26/28 Type hints Need a standard format for all upstream package archives
Future work
More modules (written by you?)
Package software for any distribution with upt 27/28 Need a standard format for all upstream package archives
Future work
More modules (written by you?) Type hints
Package software for any distribution with upt 27/28 Future work
More modules (written by you?) Type hints Need a standard format for all upstream package archives
Package software for any distribution with upt 27/28 Join me!
Code: https://framagit.org/upt Emails: [email protected], [email protected] Freenode: #upt-packaging, Steap Questions?
Package software for any distribution with upt 28/28