USENIX Association
Proceedings of the LISA 2001 15th Systems Administration Conference
San Diego, California, USA December 2Ð7, 2001
© 2001 by The USENIX Association All Rights Reserved For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. A Management System for Network-Sharable Locally Installed Software: Merging RPM and the Depot Scheme Under Solaris R. P. C. Rodgers and Ziying Sherwin – Lister Hill National Center for Biomedical Communications
ABSTRACT Efficient management of locally installed software is a recurring central theme of system administration. We report here on an experimental merger of two previously independent systems: Redhat’s RPM Package Manager (RPM), an open-source database-driven system developed by a major Linux vendor to manage software on a single host; and, an enhanced version of depot,a well-established set of conventions used to manage software that is installed on a server and shared over a network with multiple (possibly heterogeneous) clients. The combination remedies shortcomings in both systems, but to be fully effective, extensions to RPM are required, particularly to its database system. The results of this study point the way toward a second- generation network-distributed version of RPM.
Introduction Laboratory (web-based documents for which have been withdrawn from public access). Most of these Management of the operating system (OS) soft- publically-available systems were developed on UNIX ware on a computer can be time consuming; at many platforms, and attempted to support the sharing of sites, though, the amount of OS software is dwarfed installed software over multiple systems via filesys- by the amount of additional, locally installed, software tem-sharing schemes such as network file system arising from a variety of sources. Such software gener- (NFS), where clients might be using hardware and OS ally provides the services which justify the very exis- software different from that of the server. It is difficult tence of the computing facility. The proper installation to objectively measure the relative costs and benefits and maintenance of software is at the heart of system of these different approaches; however, STORE and administration, and has a major influence on the util- ASIS are much more complex than depot, and none of ity, reliability, and security of a facility. The work pre- these systems has been taken up widely outside of its sented here attempts to merge the complimentary fea- original site of invention. tures of two open source software installation and management systems: one, known as depot, is a sys- Commercially-derived systems exist as well. Sun pkgadd tem designed for managing network-shared software; introduced a software management system ( , pkgrm pkginfo the other, RPM, is designed to manage software on a , , . . .) as part of its Solaris 2 operating single host. system, using it to install Solaris itself. Functional components that can be installed independently of Prior Work other components are carved off into their own named ‘‘packages,’’ and a list is maintained of where a pack- One of the earliest attempts to attack the problem age’s files are installed. Major Linux vendors have all of creating and maintaining a network-shared local developed software packaging methods with similar software repository was the NIST depot scheme [1]. intent. These include: Redhat’s RPM Package Manager The depot conventions were widely perceived as too (RPM) [10], which is also used by the French-based complex for smaller facilities run by non-professional MandrakeSoft [11]; Ximian’s Red Carpet [12] (and administrators, leading to simplified derivatives such Redhat’s equivalent, up2date client); and Debian’s as depot-lite [2] and GNU Stow [3]. Colyer, et al. of the Package Management System [13]. Suse’s YaST inter- Andrew project at CMU offered extensions to the face appears to be more concerned with OS installa- original NIST scheme, including the notion of a soft- tion than ongoing local software installation. Caldera ware ‘‘collection’’ [4, 5]. Abbey and colleagues at the International’s Volution [14] product is claimed to man- Advanced Research Laboratory at the University of age software and other resources over a network of Texas, Austin (ARL:UT), created a set of perl scripts, (multi-vendor) Linux hosts. Of these systems, RPM is opt_depot, which facilitated use of the depot conven- likely the most widely used, due to Redhat’s substan- tions [6, 7]. Other software management schemes that tial share of the Linux market as well as to RPM being have been developed include STORE [8], the Applica- available as open source for multiple hardware plat- tion Software Installation Server (ASIS) [9], and the /pack- forms using different UNIX variants. Numerous open ages scheme employed at Los Alamos National source software applications are distributed as RPM
2001 LISA XV – December 2-7, 2001 – San Diego, CA 267 A Management System for Network-Sharable Locally Installed Software ... Rodgers & Sherwin packages. With the exception of Volution, these systems • %verifyscript (script to verify correct are concerned with management of software on a sin- functioning of the package); gle host. • %post (optional script to be executed Finally, some software applications are bundled after installation); with their own installation systems; an example is the • %preun (optional script to prepare for XPInstall system employed by the Mozilla web client. uninstallation); • %postun (optional script to be executed How RPM Works after uninstallation); and, • %files (list of files to be installed). RPM packages exist in two forms: binary-type Spec files can make use of macros, OS-specific packages contain executable code for a specific hard- conditionals, and division of the package into ware/OS combination, whereas source-type packages subpackages that can be treated differently contain the original source code used to generate the from one another. executable binary files. The RPM binary package for- 2. Place the spec file in the RPM SPECS subdirec- mat is well-defined and consists of four sections: the tory. lead (a largely abandoned file structure now used to 3. Place the source code in the RPM SOURCES identify the package), signature (the PGP and MD5 subdirectory (the %prep section of the spec file data used to validate/authenticate a package), header instructs RPM how to unpack this source and (tag-demarcated information about the package), and place it in the RPM BUILD directory; macros archive (the files constituting the package, compressed are available for dealing with common types of with GNU gzip). For a binary-type RPM package file, non-RPM source packaging such as gzipped tar the RPM command rpm -i does the following: it files). checks for the presence of any other required packages 4. Execute the rpm -ba command. This unpacks (dependency checking) and for potential conflicts (the the sources and places copies in the RPM overwriting of existing files, or the installation of the BUILD subdirectory (a helpful behavior for current or older versions of already-installed pack- software distributions which tamper with their ages); it performs any required pre-installation com- own source during the build procedures, as the mands; it installs the files associated with the current original source is left intact), changes permis- package, attempting to preserve local modifications sions as required, builds the system from made to configuration files; it performs any required source, installs the compiled and other files post-installation commands; and, it logs all of the file where specified and generates binary- and locations and other package information into the RPM source-type RPM packages, placing them in the database, which is based on Berkeley DB [15]. RPMS and SRPMS subdirectories, respectively. For a source-type RPM package file, the rpm -i This maneuver will generally have to be command does much less: it unbundles the source repeated at least twice, as the builder will want code files and specification file (see below), putting to save effort by generating the file list section the latter in the SPECS subdirectory. One then uses of the spec file by making a recursive directory the rpm -ba command to build both binary- and listing of the files installed from an earlier pass. source-type packages. As packages are built, the BUILDS subdirectory Both binary- and source-type RPM packages have gets cleaned out by RPM, but files and packages accu- to be created manually. This process employs various mulate in the SPECS, RPMS, and SRPMS directories. directories that are created when RPM is installed, Several locations require manual cleaning: the named BUILD, RPMS, SOURCES, SPECS, and SOURCES subdirectory, and a temporary directory in SRPMS, and proceeds as follows: which log files accumulate in association with failed 1. Create a RPM specification (spec) file, contain- rpm -ba commands. ing sections which address various aspects of RPM makes use of MD5 checksums to validate installing and uninstalling a package. The spec both entire packages and individual files within a file begins with a mandatory preamble consist- package (prior to and after installation), and to guide ing of tag-demarcated fields containing general the treatment of an application’s configuration files. It information about the package and its creator. It also (optionally) employs PGP [16] to create and then continues with one or more of the follow- authenticate digital signatures for packages. RPM pro- ing sections, as appropriate (using the formal vides utilities to search the RPM database to recover name, beginning with a percent sign): information about installed packages, and to easily • %prep (script to prepare for building); update and remove them. • %build (compile/build the package); The behavior of RPM can be tailored by system- • %pre (optional script to prepare for wide and user-specific initialization files. RPM is built installation); on the rpmlib library, which has an Application Pro- • %install (copy requisite files into place); grammer ’s Interface comprising over 60 different • %clean (clean the build directory tree); functions.
268 2001 LISA XV – December 2-7, 2001 – San Diego, CA Rodgers & Sherwin A Management System for Network-Sharable Locally Installed Software . . .
How Depot Works links rather than NFS to provide files to the /depot_ mount tree in that case. Henceforth in this paper, depot refers to conven- tions for software management employed at the U. S. The behavior of depot on a client is controlled by National Library of Medicine (NLM), relying upon configuration files: /depot/site controls the mounting modified versions of the ARL:UT perl scripts (origi- of files from multiple depot servers; /depot/.exclude nally known as opt_depot, and building upon the ear- prevents specified packages from being linked into lier work at NIST [1] and CMU [4, 5]). /depot/package; and, /depot/.priority controls which packages have priority for linking into /depot/{bin, The method employs the following directory tree html, ... sbin} when there are name conflicts between on a server: /depot_server/
2001 LISA XV – December 2-7, 2001 – San Diego, CA 269 A Management System for Network-Sharable Locally Installed Software ... Rodgers & Sherwin
desired). RPM automates the taking of addi- The Experimental Environment tional actions through its uninstallation scripts. The study was done using an UltraSPARC 2 as 3. Both methods attempt to document the installed depot software. At NLM, each depot package has in the server, and an UltraSPARC 2 and two Ultra- depot its root directory a manually constructed file SPARC 60 machines as clients. The machines named README.LOCAL, which includes a were operating under Solaris 2.[5-8]. We standardized header containing defined fields describing the the naming and NFS automounting schemes used by package, its author and origin, and its installer. depot as described above. At our request, Abbey and There is considerable overlap with the informa- colleagues added new features to the original ARL:UT tion contained in the RPM spec file. opt_depot scripts: the ability to mount software pack- 4. Both systems can support multiple versions of a ages on a client from multiple depot servers; and, given package, if and only if the package in improved configurability of the perl scripts. We installed question does not require the use of absolute and used depot routinely for a period of four years, file paths; with RPM, this requires use of the successfully supporting as many as two different ver- --relocate flag. sions of Solaris concurrently. At the time of writing, the depot server contained 321 shared packages and 21 The two systems compliment one another in a local packages for Solaris 2.8. Source for the SPARC- number of important respects: RPM 1. Applicability over a network. Unlike RPM, compatible version 4.0.2 was obtained from the depot is inherently designed to accommodate web site http://www.rpm.org . Eighteen lines of the use of a network. Multiple depot servers can code had to be modified to get it to compile under provide redundancy (through NFS roll-over). Solaris 2.8 using gcc 3.0. RPM was installed as a Collaboration is facilitated, as different work- shared depot package on the depot server, with the RPM groups can specialize in particular types of soft- database files placed in /depot/package/rpm_4.0.2/ ware on their own depot server, and share the vendor/var/lib/rpm. Information about all shared pack- results within their larger organization. ages is logged into this database. On each client, the 2. Dependency checking. Under NLM depot, the RPM database is installed in /var/depot/rpm_4.0.2/ installer places dependency information in the local_db, and information about local packages is README.LOCAL file that gets installed with placed there. The need for and limitations of using two the package. This manual process is error- databases is discussed below. prone. RPM employs the UNIX ldd command to To allow RPM dependency checking to operate, determine which libraries the package requires we employed a script, vpkg-provides2.sh, provided with (its ‘‘dependencies’’), and logs this information the Solaris version of RPM, which uses the information into the RPM database. At installation and unin- provided by Sun’s proprietary package database to stallation time, dependencies can be handled create entries for ‘‘virtual’’ RPM packages (there were rigorously. 564 such packages on our depot server). We then 3. Package documentation. The README.LOCAL installed a number of packages using RPM, while fol- file remains as a human-readable document lowing the depot conventions: a library (libpcap 0.4); with the installed package; various components an application depending upon that library (snort 1.7); of the RPM spec file become part of the RPM a self-standing source application (wget 1.6), and, a database record, but the spec file does not commercial pre-built binary application (netscape remain online as part of the installed package. 4.77). Unlike README.LOCAL, a spec file can con- tain procedural components (various scripts) Results/Discussion that get automatically invoked at the appropri- ate moment. Unlike the spec file, README. The RPM/depot merger experiment suggested a LOCAL includes (manual) instructions for number of technical directions for future work: host- and user-specific installation steps that a 1. The creation of ‘‘virtual’’ RPM packages from package may require to become fully func- Sun’s proprietary package format is slow, and tional, reflecting the multi-host networked the script must be rerun when additional Sun nature of depot. For example, the GNU findutils packages are installed. Ideally, Sun should use system requires that a database be built on each RPM/depot; otherwise, the script should be client host, and Sun’s StarOffice system requires improved, and wrappers created for the Sun that each user run an initialization script prior package tools to invisibly and reliably integrate to using the package. README.LOCAL also them with RPM/depot. contains transcripts of the installation proce- 2. RPM and depot functionalities should be cou- dure and copies of email correspondence with pled. RPM spec file macros could be used to other parties in connection with the software. invoke requisite depot scripts during installa- 4. User environment. Under depot, search paths tion/uninstallation, and to automate actions now are short and simple (/depot/bin, /depot/man, taken manually. Alternatively, the scripts could . . .). be invoked directly from RPM source code.
270 2001 LISA XV – December 2-7, 2001 – San Diego, CA Rodgers & Sherwin A Management System for Network-Sharable Locally Installed Software . . .
3. Capabilities of the RPM spec and depot be configured for either a local or shared depot README.LOCAL files should be merged; in package. RPM could be altered to allow the particular, RPM needs to know how to trigger package to be designated to be installed as the host- and user-specific initialization (and shared or local, and alter the installation auto- uninstallation) steps required by some software. matically. (currently, one can use the RPM Other helpful aspects of depot as used at NLM: --relocate command-line option, with inclusion of installation transcripts and related appropriate path argument, to install a shared email correspondence, and leaving a human- package as a local package, or vice versa). readable document with the installed package. 7. One of depot’s strengths is its ability to offer Transcripts of the build could be created by files for a variety of operating systems, whereas using spec file macros to capture the process in RPM is UNIX-specific. If RPM were to be made a subshell, redirecting the output to a file which to support non-UNIX target operating systems, is then incorporated into the package documen- it would have to know how to check for depen- tation. It would be ideal if the README. dencies and to build packages on the target OS. LOCAL/spec merger resulted in redundancy, This would be a considerable undertaking com- such that the README.LOCAL files could be pared to the changes suggested earlier. How- used to rapidly rebuild a corrupted RPM ever, one could still concurrently employ database, and to manually manage packages in RPM/depot for enhanced control of UNIX soft- an emergency. ware, while continuing to use depot as at pre- 4. Modifications to RPM to allow dependency sent to support the other OSs. checking over a network. RPM can only use one database at a time; either the default one, or one Conclusion supplied following the rpm command line argu- ment --dbpath. As described earlier, we Automation and standardization are two means installed RPM with two databases: one database of reducing the considerable costs of administering (on the server) containing the information for software on multiple hosts. Depot is a highly efficient shared packages (shared by all hosts over the means of managing local software, even on stand- network), and a second database for local pack- alone systems; its benefits are compounded when used ages (on each client). Most packages can be with multiple networked hosts, an environment in shared, and use of the shared package database which one can use both network-shared and local (client- will succeed much of the time, as it contains specific) depot packages. RPM is better at certain things records for most of the Solaris virtual packages such as documenting packages by database, and as well as the shared depot packages. Installa- dependency checking, but is currently designed for tion of local packages will fail more often, as use on a single machine. Our experiment in merging the local package database will not contain RPM and depot is a qualified success, in that most of information about shared packages upon which our software can be installed and used with RPM/depot local ones may depend. This problem could be without modifying RPM or depot. A fully functioning partially solved by modifying RPM, supporting RPM/depot system, however, requires slight modifica- the searching of a comma-separated list of tions to RPM source code: most importantly, to allow databases instead of just a single database (an the searching of multiple RPM databases to support enhancement that has already been requested dependency checking for local depot packages; less by other RPM users; see bugzilla request #4137 importantly, to couple execution of depot scripts to the on the Redhat web site). This is preferable to execution of RPM commands, and to support the auto- error-prone work-arounds such as cloning the matic reconfiguration of paths required when installing content of the shared database onto the local as a local depot package one that was originally databases. This solution is partial, however, designed to be shared (or vice versa). because of the way in which depot handles mul- Such modifications seem a slight price to pay in tiple versions of packages and file name colli- order to turn RPM into a network-based software man- sions, through the depot configuration files. agement tool. It would also make the system more RPM should be able to deal with these files, so attractive as a packaging system for use by other as to know how to get to the files a new pack- age needs. UNIX/Linux vendors. The existence of a single widely- 5. User-defined tags are not supported for the RPM shared system could save time for administrators, by spec file and database; such tags would be use- allowing the creation of ftp- and Web-accessible ful for local extensions to the database. archives of RPM/depot packages for Solaris (and other) 6. Other RPM enhancements could be achieved platforms, greatly reducing installation effort. more efficiently by means of modest code mod- Modifications to RPM to support non-UNIX ifications. When creating an installable RPM clients would be more complex than the ones just binary-type package, its installation path must described, and are harder to justify.
2001 LISA XV – December 2-7, 2001 – San Diego, CA 271 A Management System for Network-Sharable Locally Installed Software ... Rodgers & Sherwin
Code Availability [2] Rouillard, J. P., and R. B. Martin, ‘‘Depot-Lite: A Mechanism for Managing Software,’’ http:// The code and documentation for RPM/depot for www.usenix.org/publications/library/proceedings/ Solaris will be available at the time of the conference, lisa94/martin.html, LISA VIII, 1994. from: http://www.etg.nlm.nih.gov . [3] Glickstein, B., ‘‘GNU Stow,’’ http://www.gnu.ai. Acknowledgements mit.edu/software/stow/ , http://www.gnu.ai.mit.edu/ software/stow/manual.html . We wish to thank Jeff Johnson of Redhat for [4] Colyer, W., and W. Wong, ‘‘Depot: A Tool for invaluable assistance with installing and using RPM Managing Software Environments,’’ LISA VI, under Solaris. Jonathan Abbey of ARL:UT assisted us http://andrew2.andrew.cmu.edu/depot/depot-lisaVI- opt_depot in understanding and using his scripts, and paper.html, 1992. made helpful extensions to them at our request. Both [5] ‘‘The Depot Configuration Management Pro- provided helpful remarks about the manuscript, along ject,’’ Carnegie Mellon University, http://andrew2. with Jules Aronson of NLM and Nelson Beebe of the andrew.cmu.edu/ANDREWII/depot.html, http:// University of Utah. asg.web.cmu.edu/depot/depot.html. Funding Sources & Copyright [6] ‘‘opt_depot,’’ ARL, University of Texas at Austin, http://www.arlut.utexas.edu/csd/opt_depot/ Both authors are functioning as paid employees opt_depot.html . within a U. S. government research laboratory and [7] Abbey, J., ‘‘The Group Administration Shell and produced this work as part of their routine duties. No the GASH Network Computing Environment,’’ additional funding was involved. As a work produced LISA VIII, http://www.arlut.utexas.edu/csd/gash_ at government expense, this text is placed in the public docs/lisa_paper/paper.html, September, 1994. domain and can not be copyrighted. [8] Bakken, S. S., A. Christensen, T. Egge, and A. Biographical Notes H. Juul, ‘‘STORE,’’ Norwegian University of Science and Technology, http://www.pvv.unit.no/ R. P. C. Rodgers ([email protected]) works ˜arnej/store/storedoc.html . in biomedical informatics at the Lister Hill National [9] Defert, P., S. Gouache, A. Peyrat, and I. Reguero, Center for Biomedical Communications (LHNCBC), ‘‘ASIS User’s and Reference Guide, Version where he heads the Emerging Technologies Group. He 3.95,’’ http://consult.cern.ch/writeups/asis/node1. received a B.A. From Harvard College in 1972, a html, CERN, 1997. M.D. from the University of Utah College of Medicine [10] Bailey, E. C., Maximum RPM, SAMS Publishing in 1976, and postdoctoral training from the University http://www.rpm.org/max-rpm/index.html; http:// of London, University of Louvain, the National Can- www.rpmdp.org/rpmbook, 1997. cer Institute, and the University of California, San Francisco (UCSF). He served on the faculty at UCSF [11] Bégnis, C., G. Cottenceau, G. Lee, and T. Vig- prior to joining LHNCBC, a research arm of the U. S. naud, Mandrake RPM HOWTO, vol. 1.1, http:// National Library of Medicine (NLM). At NLM he www.linux-mandrake.com/en/howtos/mdk-rpm/ . became an early and active exponent of the World [12] Ximian, ‘‘Red Carpet,’’ http://www.ximian.com/ Wide Web, creating and running NLM’s web services products/ximian_red_carpet/ . for the first two years of their existence. He has partic- [13] Debian, ‘‘Package Management System,’’ http:// ipated in a number of IETF working groups, and www.debian.org/doc/FAQ/ch-pkg_basics.html, http:// served as a founding member of the International www.debian.org/doc/packaging-manuals/developers- World Wide Web Conference Committee and found- reference/ . ing chair of the NSF/NCSA World Wide Web Federal [14] Caldera International, ‘‘Volution,’’ http://www. Consortium. caldera.com/products/volution/ . Ziying Sherwin ([email protected]) received [15] Sleepycat Software Inc., Berkeley DB, New Rid- a B.S. in Computing & Engineering from Zhejiang ers Publishing, Indianapolis, 2001. University in 1996, and a M.S. in Computer & Infor- [16] Garfinkel, S., PGP: Pretty Good Privacy, First mation Science from the University of Delaware in Edition, December, 1994. 1999. She has worked Bell for Atlantic, and joined the Emerging Technologies Group at LHNCBC in 2000. References [1] Manheimer, K., B. Warsaw, S. N. Clark, and W. Rowe, ‘‘The Depot: A Framework for Sharing Software Installation Across Organizational and UNIX Platform Boundaries,’’ LISA IV, http:// www.forwiss.uni-passau.de/archive/marchiv/ systemverwaltung.html, 17-19 October, 1990.
272 2001 LISA XV – December 2-7, 2001 – San Diego, CA