Software Deployment Packaging, Installation, Updates Peter Libič, Pavel Parízek, Tomáš Kalibera DISTRIBUTED SYSTEMS RESEARCH GROUP http://dsrg.mff.cuni.cz CHARLES UNIVERSITY PRAGUE Faculty of Mathematics and Physics What is software deployment • Process of making a software system available to the user . Packaging . Distribution . Installation . Updates . Uninstall Key challenges Key challenges • Respecting standards and conventions of a target platform • Dependency hell • Configuration and user files • Automation • Robustness Respecting standards and conventions • File system hierarchy . Putting binaries and configuration files at proper place • Unix/Linux: /usr/bin, /opt, /etc, … • Windows: C:\Program Files, … • Names of packages and files . Examples • linux-2.6.26.tar.gz • kernel-2.6.25-14.fc9.i586.rpm • Use of the same mean of deployment as . Target system/platform . Required third-party libraries • Easier resolving of dependencies File system hierarchy: Unix • Binaries: . /bin, /sbin, /usr/bin • Libraries: . /lib, /usr/lib, /var/lib • Configuration: . /etc (general), /home/XXX (user specific) • Prefixes . /var (variable content, RW) . /usr (secondary hierarchy for apps, RO) . /opt/XXX (zero installed packages) . /home (user directories) • Filesystem Hierarchy Standard (FHS) - Linux . Very detailed: core utilities, applications . Additional conventions by distributions File system hierarchy: Windows • Binaries and program data: . C:\Program Files\XXX (+special semantics of some subdirs by MS) • Libraries: . C:\Windows\System32 • Configuration (user specific & general) . C:\Documents and Settings\XXX • Windows registry . Database of configuration and data File system hierarchy: Mac OS/X • Applications . /Applications/XXX (complete application distribution in a single archive - image file .dmg) • Core system . /System . /Library – configuration • Home directories . /Users/XXX Dependency hell: how to install, upgrade with… • Conflicting dependencies . Different software systems may depend on different versions of shared libraries, which may not necessarily be able to co-exist • Many dependencies, long dependency chains, circular dependencies . Hard to resolve • Software requires a small part of a big library/many libraries . All have to be downloaded, installed, including dependencies . All have to be ported, kept up to date… Fighting the dependency hell • Software packages with reasonable meta- data . Versions – i.e. major and minor . Dependencies described at good granularity • Manual . Tweaking of source code (works for conflicts) . Tracking and installation of all dependencies by hand • Automated . Static linking of dependencies ( huge binary), independent software . Repository-based package management systems (a “single” repository for an OS) • The problem is solved by distribution maintainers Profound instances of dependency hell • JARs . All linking is dynamic in Java . Extremely difficult to fight by end users • Hierarchy of class loaders • Applications run in a single process (JVM), which has a (single) “classpath” • RPMs . Packages from different distributions with conflicting dependencies, package names Profound instances of dependency hell (mostly resolved by now): DLLs • Over-use of shared libraries • Overwriting libraries by applications, no versioning • Central COM registry • NOW . processes can have their own (versions) of libraries . Dependencies can be specified . Registration-free COM Configuration and user files • Update . Issue: what to do with configuration files • User may have modified the original configuration . Solutions • Preserve user’s configuration and save the new configuration under a different file name . User has to update the configuration manually (e.g. to a new file format) • Uninstall . Issue: what to do with configuration and user files • User files: documents, saved files from games, … . Solutions • Preserve them (for re-install in future), backup, remove • Usually it is up to the user to decide Other challenges of software deployment • Automation . One-step installation process • Using a single command . Automated updates • downloading, installation • Robustness . Target system has to be left in a consistent state • No partial installation • Use of transactions Means of software deployment Means of software deployment • Source code • Zero installation • Package management system • Application-specific installer Source code • Advantage . Allows installation of selected features only and high degree of optimization • Depending e.g. on the configuration of target system, user preferences, available libraries and CPU architecture • Drawback . User has to perform some operations manually • download, configuration, compilation • Typical for GNU software . “./configure && make && make install” (Autotools) Source code – distribution of updates • Two options of distribution . Patch • Based on “diff” between old and new version • Has to be applied manually to the old source code . Archive with new version • Directory with unpacked old source code is removed • Update process 1.Uninstall of old version (via “make uninstall”, or “rm –rf /opt/XXX” for self-contained installations) 2. Application of patch or unpacking of archive 3. Compilation and installation of a new version • “make; make install” Zero installation • No explicit installation . Copying of files to target location . Manual “post-installation” configuration • Setting some paths, … • Typical for Java programs . Distributed as a set of binary files (JAR, class) and configuration files (xml, properties) in an archive . Archive can be created by Ant or Maven • Ant tasks: jar, zip Package management system • Collection of tools for automated and consistent processing of software packages • Key concepts . Package • Means of distribution of software and metadata • Metadata . Version, description, list of dependencies, … . Local package database • Information about installed packages (metadata) . Software repository • Remote storage of all available packages Package management system – tasks • Downloading of a package • Identification of dependencies (required packages) • Installation . Including all required dependencies • Updating . Including updates to (some) dependencies • Uninstall . Including some dependencies (not used by other programs) • Querying of information about packages . Installed on a system, available in a repository • Getting sources for a package . For an installed binary package, find and download the (exact) sources it was built from • Building a package . For a (specially structured and annotated) tree, build a package Package management system – examples • System-level package managers . Used in major Linux distributions • RPM (Fedora, SUSE) • DEB (Debian, Ubuntu) • Ports (Gentoo – Portage, FreeBSD, MacOS X - macports ) . Windows installer (MSI files) . “App stores” • Android Market • Apple App Store • Application-level package managers . CPAN (Perl – Comprehensive Perl Archive Network) . XPI (Firefox, Thunderbird) . Eclipse plugins . R project (CRAN) Example System: APT Apt package manager • Installation . “apt-get install firefox” . “-s” option – only print what would be done • Removal . “apt-get remove firefox” • Upgrade . “apt-get upgrade”, “apt-get dist-upgrade” • Update of local cache of meta-data . “apt-get update” Apt package manager: source packages • Getting sources of a package . “apt-get source firefox” . Downloads • Original package code (tar.gz archive) • Patch for distribution (added meta-data, bugfixes) • Build meta-data • Building retrieved sources . Automatic • on download or by “dpkg-buildpackage” . Dependencies for building • Installed automatically:“apt-get build-dep firefox” Apt package manager: Queries • Which package does an installed file come from ? . “dpkg -S stdio.h” (full path or name only) • Which packages are installed ? . “dpkg -l” • Fulltext search of descriptions of all available (not only installed) packages . “apt-cache search browser” • Which package (even uninstalled) would contain a given file ? . “apt-file search stdio.h” . File lists are cached locally if apt-file is used Apt package manager: Distributions • Debian GNU/Linux (DEB packages) • Ubuntu • Conectiva (RPM packages) • Redhat • Fedora • Suse • And many others… Back to package managers overview Package managers – distribution of updates • Two options . Package with a complete new version • Old version is “removed/uninstalled” . Diff package • Not so common, but still supported by some (delta RPM – Suse, openSuse, Fedora) • The issue is an efficient diff algorithm for compressed binary archives Application-specific installer • Installation program is a part of the distributed package • Typical for . Older Windows applications • InstallShield . Large proprietary applications • Often do not respect standards and conventions • Oracle Database, Borland Together, … • Windows today . Installers usually based on Windows Installer framework by Microsoft . Support in Visual Studio.. Package manager vs. installer • Package management system . Part of the operating system • Software packages are not distributed with an installer . Single database for all packages . Single package format • Application-specific installer . Each application is distributed with its own installer . No single database for all installed software • Some information are stored in registry . Multiple package formats RedHat Package Manager/Packages (RPM) RedHat Package Manager (RPM) • Supports . Installation • rpm -ivh [package file] • Example: rpm –ivh