Software Deployment Packaging, Installation, Updates

Home , APT (software), Dependency hell, Dpkg, MacPorts, Open Build Service, Portage (software)

Peter Libič, Pavel Parízek, Tomáš Kalibera DISTRIBUTED SYSTEMS RESEARCH GROUP http://dsrg.mff.cuni.cz

CHARLES UNIVERSITY PRAGUE Faculty of Mathematics and Physics What is software deployment

• Process of making a software system available to the user . Packaging . Distribution . Installation . Updates . Uninstall Key challenges Key challenges

• Respecting standards and conventions of a target platform • Dependency hell • Configuration and user files

• Automation • Robustness Respecting standards and conventions

• File system hierarchy . Putting binaries and configuration files at proper place • Unix/Linux: /usr/bin, /opt, /etc, … • Windows: C:\Program Files, …

• Names of packages and files . Examples • linux-2.6.26.tar.gz • kernel-2.6.25-14.fc9.i586.rpm

• Use of the same mean of deployment as . Target system/platform . Required third-party libraries • Easier resolving of dependencies File system hierarchy: Unix

• Binaries: . /bin, /sbin, /usr/bin • Libraries: . /lib, /usr/lib, /var/lib • Configuration: . /etc (general), /home/XXX (user specific) • Prefixes . /var (variable content, RW) . /usr (secondary hierarchy for apps, RO) . /opt/XXX (zero installed packages) . /home (user directories) • Filesystem Hierarchy Standard (FHS) - Linux . Very detailed: core utilities, applications . Additional conventions by distributions File system hierarchy: Windows

• Binaries and program data: . C:\Program Files\XXX (+special semantics of some subdirs by MS) • Libraries: . C:\Windows\System32 • Configuration (user specific & general) . C:\Documents and Settings\XXX • Windows registry . Database of configuration and data File system hierarchy: Mac OS/X

• Applications . /Applications/XXX (complete application distribution in a single archive - image file .dmg) • Core system . /System . /Library – configuration • Home directories . /Users/XXX Dependency hell: how to install, upgrade with…

• Conflicting dependencies . Different software systems may depend on different versions of shared libraries, which may not necessarily be able to co-exist • Many dependencies, long dependency chains, circular dependencies . Hard to resolve • Software requires a small part of a big library/many libraries . All have to be downloaded, installed, including dependencies . All have to be ported, kept up to date… Fighting the dependency hell

• Software packages with reasonable meta- data . Versions – i.e. major and minor . Dependencies described at good granularity • Manual . Tweaking of source code (works for conflicts) . Tracking and installation of all dependencies by hand • Automated . Static linking of dependencies ( huge binary), independent software . Repository-based package management systems (a “single” repository for an OS) • The problem is solved by distribution maintainers Profound instances of dependency hell

• JARs . All linking is dynamic in Java . Extremely difficult to fight by end users • Hierarchy of class loaders • Applications run in a single process (JVM), which has a (single) “classpath” • RPMs . Packages from different distributions with conflicting dependencies, package names Profound instances of dependency hell (mostly resolved by now): DLLs

• Over-use of shared libraries • Overwriting libraries by applications, no versioning • Central COM registry • NOW . processes can have their own (versions) of libraries . Dependencies can be specified . Registration-free COM Configuration and user files

• Update . Issue: what to do with configuration files • User may have modified the original configuration . Solutions • Preserve user’s configuration and save the new configuration under a different file name . User has to update the configuration manually (e.g. to a new file format)

• Uninstall . Issue: what to do with configuration and user files • User files: documents, saved files from games, … . Solutions • Preserve them (for re-install in future), backup, remove • Usually it is up to the user to decide Other challenges of software deployment

• Automation . One-step installation process • Using a single command . Automated updates • downloading, installation

• Robustness . Target system has to be left in a consistent state • No partial installation • Use of transactions

Means of software deployment Means of software deployment

• Source code • Zero installation • Package management system • Application-specific installer Source code

• Advantage . Allows installation of selected features only and high degree of optimization • Depending e.g. on the configuration of target system, user preferences, available libraries and CPU architecture • Drawback . User has to perform some operations manually • download, configuration, compilation

• Typical for GNU software . “./configure && make && make install” (Autotools) Source code – distribution of updates

• Two options of distribution . Patch • Based on “diff” between old and new version • Has to be applied manually to the old source code . Archive with new version • Directory with unpacked old source code is removed • Update process 1.Uninstall of old version (via “make uninstall”, or “rm –rf /opt/XXX” for self-contained installations) 2. Application of patch or unpacking of archive 3. Compilation and installation of a new version • “make; make install” Zero installation

• No explicit installation . Copying of files to target location . Manual “post-installation” configuration • Setting some paths, …

• Typical for Java programs . Distributed as a set of binary files (JAR, class) and configuration files (xml, properties) in an archive . Archive can be created by Ant or Maven • Ant tasks: jar, zip

Package management system

• Collection of tools for automated and consistent processing of software packages

• Key concepts . Package • Means of distribution of software and metadata • Metadata . Version, description, list of dependencies, … . Local package database • Information about installed packages (metadata) . Software repository • Remote storage of all available packages Package management system – tasks

• Downloading of a package • Identification of dependencies (required packages) • Installation . Including all required dependencies • Updating . Including updates to (some) dependencies • Uninstall . Including some dependencies (not used by other programs) • Querying of information about packages . Installed on a system, available in a repository

• Getting sources for a package . For an installed binary package, find and download the (exact) sources it was built from • Building a package . For a (specially structured and annotated) tree, build a package Package management system – examples

• System-level package managers . Used in major Linux distributions • RPM (Fedora, SUSE) • DEB (Debian, Ubuntu) • Ports (Gentoo – Portage, FreeBSD, MacOS X - macports ) . Windows installer (MSI files) . “App stores” • Android Market • Apple App Store • Application-level package managers . CPAN (Perl – Comprehensive Perl Archive Network) . XPI (Firefox, Thunderbird) . Eclipse plugins . R project (CRAN) Example System: APT Apt package manager

• Installation . “apt-get install firefox” . “-s” option – only print what would be done • Removal . “apt-get remove firefox” • Upgrade . “apt-get upgrade”, “apt-get dist-upgrade” • Update of local cache of meta-data . “apt-get update” Apt package manager: source packages

• Getting sources of a package . “apt-get source firefox” . Downloads • Original package code (tar.gz archive) • Patch for distribution (added meta-data, bugfixes) • Build meta-data • Building retrieved sources . Automatic • on download or by “dpkg-buildpackage” . Dependencies for building • Installed automatically:“apt-get build-dep firefox” Apt package manager: Queries

• Which package does an installed file come from ? . “dpkg -S stdio.h” (full path or name only) • Which packages are installed ? . “dpkg -l” • Fulltext search of descriptions of all available (not only installed) packages . “apt-cache search browser” • Which package (even uninstalled) would contain a given file ? . “apt-file search stdio.h” . File lists are cached locally if apt-file is used Apt package manager: Distributions

• Debian GNU/Linux (DEB packages) • Ubuntu • Conectiva (RPM packages) • Redhat • Fedora • Suse • And many others… Back to package managers overview Package managers – distribution of updates

• Two options . Package with a complete new version • Old version is “removed/uninstalled” . Diff package • Not so common, but still supported by some (delta RPM – Suse, openSuse, Fedora) • The issue is an efficient diff algorithm for compressed binary archives Application-specific installer

• Installation program is a part of the distributed package • Typical for . Older Windows applications • InstallShield . Large proprietary applications • Often do not respect standards and conventions • Oracle Database, Borland Together, … • Windows today . Installers usually based on Windows Installer framework by Microsoft . Support in Visual Studio.. Package manager vs. installer

• Package management system . Part of the operating system • Software packages are not distributed with an installer . Single database for all packages . Single package format

• Application-specific installer . Each application is distributed with its own installer . No single database for all installed software • Some information are stored in registry . Multiple package formats RedHat Package Manager/Packages (RPM) RedHat Package Manager (RPM)

• Supports . Installation • rpm -ivh [package file] • Example: rpm –ivh foobar-1.2.1.i386.rpm . Uninstall • rpm -e [package name] • Example: rpm –e foobar . Updating • rpm -Uvh [package file] . Querying • rpm -q [package name] . …

• Used in several Linux distributions . Fedora, RHEL, SUSE Linux, CentOS, Mandriva, … • Part of Linux Standard Base (LSB) RPM package – contents

• Signature . For verification of integrity

• Header (meta-data) . Version . Description . List of dependencies . …

• Payload . Archive with files How to create a RPM package

1) Prepare the archive with source code . Example: foo-1.2.tar.gz

2) Write a spec file . Input for the rpm tool • Tells RPM how to compile the software and create the package . Example: foo.spec

3) Log in as root

4) Copy the archive with source code to the directory /usr/src/redhat/SOURCES . RPM assumes that this directory is used

5) Run the rpmbuild -ba [spec file] command . Example: rpmbuild -ba foo.spec . Creates two RPM packages • /usr/src/redhat/SRPMS/foo-1.2-1.src.rpm • /usr/src/redhat/RPMS/i386/foo-1.2-1.i386.rpm Example of spec file

Summary: Foo in /usr/src/redhat/SOURCES Name: foo Version: 1.2 Release: 1 header Source0: %{name}-%{version}.tar.gz License: GPL Group: Misc %description The Foo program does nothing interesting %prep %setup -q unpacks the source archive %build ./configure make all files are installed as owned by root %install make install %files List of files to be included %defattr(-,root,root) /usr/local/bin/foo in the RPM package %doc COPYING README files tagged as documentation %config /etc/foo files tagged as configuration Creation of RPM packages – issues

• Software is installed on the build system during preparation of the RPM package . Solution: build root

• Developer has to be logged in as root . Solution: use different set of directories than /usr/src/redhat Build root

• Idea . Dummy directory tree is used for all installed files

• Example /usr/src/redhat/BUILD … Group: Misc BuildRoot: %{_builddir}/%name-root %description … %install rm -rf $RPM_BUILD_ROOT make DESTDIR=$RPM_BUILD_ROOT install %clean rm -rf $RPM_BUILD_ROOT %files called at the end … Different set of directories

• Idea: change the %_topdir setting . Create a file .rpmmacros in $HOME and add e.g. the following line to it: • %_topdir /home/username/rpm

• The /home/username/rpm directory is used instead of /usr/src/redhat . Soure code archive has to be copied into the /home/username/rpm/SOURCES directory Creation of RPM packages – advanced topics

• Dependencies • Virtual capabilities • Installation scripts Dependencies among RPM packages

• Required package . Add requires: [package name] into the spec file . Example • requires: bar

• Required package of a specific version only . Use requires: [package name] = [version number] . Example • requires: bar = 2.1.6

• Required package of a specific version or higher . Use requires: [package name] >= [least version] . Example • requires: bar >= 2.1 Virtual capabilities

• Basic idea . Same functionality can be provided by one from a set of packages . Arbitrary name of a capability can be used in the specification of provision • The provides clause • Default value is the name of a package

• Example . Tomcat package • requires: java . Packages for J2SE: Sun Java, GCJ + Classpath, … • provides: java Installation scripts

• Motivation . Non-trivial operations have to be performed before/after installation • Update of runlevel information • Automated configuration • …

• Special sections in spec file . Pre-installation: %pre section . Post-installation: %post section . Each section contains a shell (bash) script

• Example %post /sbin/adduser apache # modify firewall configuration /sbin/service iptables restart Open Build Service

• Open source “web service” • Easy and automated rpm/deb package building • https://build.opensuse.org/ Ports

• Package manager family • Based on (automated) source code building • Originally in BSD systems . Directories with makefiles and patches Portage

• Package management system for Gentoo

• Key characteristics . Building of packages from source code • Optimization to HW and available libraries . Packages defined via shell scripts (ebuilds) • Similar to RPM spec files . Organization of packages into categories • e.g. app-editors, dev-perl, … Ebuild

• Shell script (bash) . Description of the package . Functions and instructions for • download, unpacking, compilation, installation, …

• Created for each version of the package . Naming: pkg-ver[_suf][-r#].ebuild

Gentoo-specific predefined suffixes: revision number alpha, beta, pred2, rc3 -r1, -r2, …

. Example: eject-2.1.5-r1.ebuild

• Stored in /usr/portage/category/pkgname Ebuild – example (part 1)

DESCRIPTION="A command to eject a disc from the CD-ROM drive“ HOMEPAGE="http://eject.sourceforge.net/" SRC_URI="http://www.ibiblio.org/pub/Linux/utils/disk-management/${P}.tar.gz"

LICENSE="GPL-2“ SLOT="0“ KEYWORDS="~alpha amd64 arm hppa ia64 m68k mips ~ppc ~ppc64 s390 sh sparc x86“ IUSE="nls“

DEPEND="!virtual/eject“ PROVIDE="virtual/eject“

S=${WORKDIR}/${PN}

various shell functions ... Ebuild – example (part 2)

... header and definition of variables

src_unpack() { unpack ${A} cd "${S}“ epatch "${FILESDIR}/${PN}"-2.0.13-xmalloc.patch }

src_compile() { econf $(use_enable nls) || die invokes ./configure emake || die } uses automake src_install() { emake DESTDIR="${D}" install || die "make install failed“ dodoc ChangeLog README PORTING TODO AUTHORS NEWS PROBLEMS } Variables in ebuild

• Package name and version: P • Package name: PN • Source files: A • Root build directory: WORKDIR • Temporary build directory: S • Temporary installation directory: D . Image is then copied to the real installation directory • List of use flags used in the ebuild: IUSE

Functions in ebuilds

• Control of building and installation process . src_unpack • unpack sources, run autotools . src_compile • configure and build package . src_install • Install into the directory specified by ${D} variable . pkg_config • Initial configuration . ... To be written by the developer Functions in ebuilds – cont.

• Helper functions . use feature • returns true or false . use_with feature • returns --with-feature or --without-feature . use_enable feature • Returns --enable-feature or --disable-feature . …

Provided by the ebuild system USE flags

• Mechanism for definition of optional features that should be included in (or excluded from) packages during building . e.g. support for DVD, mp3, …

• Reflected in compile-time options by ebuilds . USE flag “feature” is translated to ./configure --with-feature

Specification of dependencies

• via DEPEND variable . Dependencies on specific package versions are supported

category has • Example to be always included DEPEND=“virtual/kde X11-libs/gtk+-2 >=sys-libs/glibc-2.2.5” Emerge tool

• Command-line utility . Primary user interface for Portage

• What it does . Executes ebuilds for individual packages • building, installation, … . Maintains database of installed packages . Automated resolving of dependencies Building from sources vs. binary packages

• Dependencies . Sources: flexibility via compile-time options • Optimization for libraries available on target system . Binaries: several versions for typical cases • Exponential number of possible configurations

• Installation time . Sources: compilation is (very) slow • Precompiled binary packages available for large programs . LibreOffice, Firefox, … . Binaries: customization often not possible Links

• Other . http://en.wikipedia.org/wiki/Software_deployment . http://en.wikipedia.org/wiki/Windows_Installer . http://en.wikipedia.org/wiki/DLL_hell . http://en.wikipedia.org/wiki/Dependency_hell . http://www.informit.com/articles/article.aspx?p=101731 (Windows directory layout) . http://www.ibm.com/developerworks/library/l-rpm1/ . Package management cheat-sheet • http://distrowatch.com/dwres.php?resource=package-management • RPM . http://wiki.rpm.org/Docs . http://docs.fedoraproject.org/drafts/rpm-guide-en/ • Portage . http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml