Taxonomy of Package Management in Programming Languages and Operating Systems Hisham Muhammad Lucas C. Villa Real Michael Homer Kong Inc. IBM Research Victoria University of Wellington - [email protected] [email protected] Wellington, New Zealand [email protected] Abstract for node.js [3], a JavaScript environment. On a Mac system, Package management is instrumental for programming lan- the typical way to install command-line tools such as npm guages and operating systems, and yet it is neglected by both is via either Homebrew [4] or MacPorts [5], the two most areas as an implementation detail. For this reason, it lacks the popular general-purpose package managers for macOS. This same kind of conceptual organization: we lack terminology is not a deliberately contrived example; it is the regular way to classify them or to reason about their design trade-offs. In to install development modules for a popular language in a this paper, we share our experience in both OS and language- modern platform. specific package manager development, categorizing families The combinations of package managers change as we move of package managers and discussing their design implications to a different operating system or use a different language. beyond particular implementations. We also identify possibil- Learning one’s way through a new language or system, nowa- ities in the still largely unexplored area of package manager days, includes learning one or more packaging environments. interoperability. As a developer of modules, this includes not only using pack- age managers but also learning to deploy code using them, Keywords package management, operating systems, module which includes syntaxes for package specification formats, de- systems, filesystem hierarchy pendency and versioning rules and deployment conventions. Simply ignoring these environments and managing modules 1 Introduction and dependencies by hand is tempting, but the complexity Package managers are programs that map relations between of heterogeneous environments and keeping track of depen- files and packages (which correspond to sets of files), and dency updates can become burdensome — all these package between packages (dependencies), allowing users to perform managers were created to solve practical problems which maintenance of their systems in terms of packages rather than the developer would have to otherwise directly handle, after at the level of individual files. Package management is an all. Another alternative that is often proposed, especially by area that lies somewhere in the border between programming users of operating systems that feature a system-provided languages and operating systems: packaging is a step that package manager (as is the case of most Linux distributions), sits after a language’s build process, and before an operating is to avoid using multiple package managers and use a single system’s component installation. For this reason, it seems general-purpose package manager. This is, of course, as much to be overlooked by both fields as an implementation issue. of a solution as trying to make everyone agree on a single In the meantime, package management keeps growing in programming language — one of many analogies between complexity. New languages, new deployment models, and package management and programming languages. The result new portability requirements all give rise to new package is that the ecosystem is not getting any simpler, and at first management systems. Further, this is not simply a matter of glance it seems that package management is indeed a largely competing implementations: modern complex environments unsolved problem. often require several package managers to be used in tandem. However, maybe the statement “package management is For example, when writing JavaScript web applications on an unsolved problem” simply does not make sense, and is a Mac environment, a developer may require using Bower akin to saying that “programming languages are an unsolved [1], a package manager for client-side JavaScript compo- problem”. In the programming languages world we accept nents. Bower is installed using npm [2], a package manager that the multitude of languages is a given. Beyond that, we understand that there are families of languages with different PLOS ’19, October 27, 2019, Huntsville, ON, Canada paradigms, with well-known tradeoffs. We also accept that © 2019 Association for Computing Machinery. there is room for domain-specific languages (DSLs) and for This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published general-purpose languages. Most importantly, we know how in PLOS ’19: Workshop on Programming Languages and Operating Systems, to set boundaries for each language and how to make DSLs October 27, 2019, Huntsville, ON, Canada, https://doi.org/10.1145/3365137. and general-purpose languages interact. Most existing pack- 3365402. age management systems, however, are still oblivious to the PLOS ’19, October 27, 2019, Huntsville, ON, Canada Hisham Muhammad, Lucas C. Villa Real, and Michael Homer fact that they exist as part of a larger ecosystem, with parts of path that conforms to Python’s default directory for locally- it handled by other package managers. installed modules (e.g. /usr/lib/python3.7/site-pack- In this paper we draw on our own unique combination of ages/), whose definition predates the introduction of pip. experiences on all sides of this topic: developing a system- Database-oriented designs are often chosen when the package wide package manager for a Linux distribution [6, 7], creating manager needs to accomodate a pre-existing directory struc- a language-specific package manager [8], and integrating ture. If Python packages have modules with the same name, system and language-specific package managers [7], as well clashes may occur. In a filesystem-oriented design, such as for as being simply developers and end users of other software. example that of RubyGems, this problem would not happen. By building a taxonomy for package management and sharing Each package has its own subtree under a versioned directory, our experiences with package management development in and the rubygems.rb module, part of the default installation both the programming language and operating system spaces, of Ruby, takes care of finding the appropriate files when mod- we aim in this paper to frame the design choices, layers, ules are loaded with the require function. Figure 1 lists more and trade-offs attendant to package managers for developers, examples of package managers and their classification. maintainers, and users. The major trade-off between the filesystem-oriented and the database-oriented approaches is whether applications 2 Paradigms of package management: should be aware of the file structure defined by the man- filesystem-oriented vs. database-oriented ager or whether the manager should adapt to the file structure defined by applications. This affects how the manager tracks It is a typical didactic device to organize the landscape of the mapping of files and how applications are configured to programming languages into paradigms, such as imperative, find their resource files. functional, object-oriented, and so on. The paradigms a lan- In filesystem-oriented managers the mapping of files to guage is categorized into inform users about particular design packages is simple. File conflicts are naturally avoided by choices, and with these choices come design trade-offs. In storing files of different packages in separate subtrees. Ver- the world of package managers, we can also identify general sioning conflicts between variants of the same package can paradigms, by looking at their core concepts and design trade- also be handled via the tree structure. The structure also be- offs. Package managers map between files and packages, and comes more transparent to users, which can simplify their between packages and their dependencies, allowing users to experience. The run-time lookup of files by applications, how- perform maintenance of their systems in terms of packages ever, can be complicated, if they are oblivious to the structure rather than at the level of individual files. The central design defined by the package manager. Applications must either choice in a package manager, therefore, is how to perform agree beforehand to this structure (which might be an option those mappings. in domain-specific environments), or the package manager There are two approaches on how to map files to packages: has to do extra work to configure them to use the structure, the mapping can be based on the hierarchical directory struc- such as setting configuration options or environment variables, ture where the files reside, or can be separate. As this choice or in the worst case, patching them. embodies a series of trade-offs and is the single decision that Conversely, in database-oriented managers the mapping affects the design and implementation of a package manager of files to packages is more complicated. Applications may the most, we identify these as two paradigms of package install files wherever they please, and the package manager management. When the mapping is internal to the file hierar- needs to keep track. This includes handling potential conflicts chy structure, we say that package management is filesystem- if two packages want to use the same pathname. Database- oriented1. When it is external to the hierarchy of files being oriented systems will usually report on these conflicts and managed, the mapping needs to be stored elsewhere. We say forbid them. It is up to the integrator (such as a distribution in these cases that management is database-oriented. Most developer) who is building packages to resolve the conflict package managers for Linux distributions, such as RPM and somehow. Also, the package manager needs to verify that the dpkg/APT, are database-oriented. Filesystem-oriented pack- database and the contents of the filesystem remain in sync, age management is more often seen in language-specific pack- which is trivial in the filesystem-oriented approach.