Overhauling Amd for the '00s: A Case Study of GNU Autotools Erez Zadok Stony Brook University [email protected] Abstract suring that software can build cleanly and run identically on many systems include the following: The GNU automatic software configuration tools, Au- toconf, Automake, and Libtool, were designed to help ¢ Asking users to manually configure a package prior the portability of software to multiple platforms. Such to compilation by editing a header file to turn on autotools also help improve the readability of code and or off package features or to specify services avail- speed up the development cycle of software packages. able from the platform on which the package will In this paper we quantify how helpful such autotools are be run. This process required intimate knowledge to the open-source software development process. We of the system on which the package (e.g., C-News) study several large packages that use these autotools and was to be built. measure the complexity of their code. We show that total ¢ Trying to achieve portability using CPP macros code size is not an accurate measure of code complex- and nested #ifdef statements. Such code results ity for portability; two better metrics are the distribution in complex, system-specific, deeply-nested CPP of CPP conditionals in that code and the number of new macros which are hard to maintain. special-purpose Autoconf macros that are written for the ¢ Using Imake [2], a system designed specifically package. for building X11 applications. Imake defines We studied one package in detail—Am-utils, the frozen configurations for various systems. How- Berkeley Automounter. As maintainers and developers ever, such static configurations cannot account for of this package, we tracked its evolution over ten years. local changes made by administrators. This package was ported to dozens of different platforms ¢ Using Metaconfig [11], developed primarily for and in 1997 was converted to use GNU autotools. We building Perl. This system executes simple tests show how this conversion (autotooling) resulted in a dra- similar to Autoconf, but users are often asked to matic reduction in code size by over 33%. In addition, confirm the results of these tests or to set the re- the conversion helped speed code development of the sults to the proper values. Metaconfig requires too Am-utils package by allowing new features and ports to much user interaction to select or confirm detected be integrated easily: for the first year after the conversion features and it cannot be extended as easily as Au- to GNU autotools, the Am-utils package grew by over toconf. 70% in size, adding many new features, and all without increasing the average code complexity. GNU Autoconf [5] solves the above problems by pro- viding canned tests that can dynamically detect various features of the system on which the tests are run. By 1 Introduction actually testing a feature before using it, Autoconf and its sister tools Automake [6] and Libtool [7] can build Large software packages, especially open-source (OSS) packages portably without user intervention. These au- ones, must be highly portable so as to maximize their use tomated software configuration tools (autotools [10]) can on as many systems as possible. Past techniques for en- run on numerous systems. Autotools work identically re- ¡ Appears in proceedings of the 2002 Annual Usenix Technical Con- gardless of the OS version, any local changes that admin- ference, FreeNIX track. istrators installed on the system, which system software 1 packages were installed or not, and which system patches on the name of the compiler (GNUC) rather than were installed. directly testing for that feature’s existence. The rest of this paper is organized as follows. Section 4. Shared libraries: Many packages need to build and 2 explains the motivation for automated software config- use shared or static libraries. Such packages of- uration tools. In section 3 we explain how GNU Auto- ten support shared libraries only on a few systems conf and associated tools work, explore their limitations, (e.g., Tcl before it was autotooled), because of dif- and describe how we used these tools. Section 4 eval- fering shared library implementations. Frequent use uates several large OSS packages and details the use of of non-shared (static) libraries results in duplicated autotools in the Am-utils package. We conclude in Sec- code that wastes disk space and memory. tion 5. 5. Human errors: Manually-configured software is more prone to human errors. For example, the first port of Amd to Solaris on the IA32 platforms 2 Motivation copied the static configuration file from the SPARC platform, incorrectly setting the endianness to big- When software packages grow large and are required to endian instead of little-endian. work on multiple platforms, they become more difficult 6. Novice and overworked administrators: With a to maintain without automation. We spent several years rapidly growing user base and the growth of the In- maintaining Amd and Am-utils, as well as fixing, port- ternet, the average expertise of system administra- ing, and developing other packages. During that time tors has decreased. Overworked administrators can- we noticed how difficult it was to maintain and port such not afford to maintain and configure many packages packages and that led us to convert Amd to use autotools. manually. As a result of the conversion, we noticed that Am-utils became easier to maintain and port. We therefore set out Converting OSS packages to use GNU autotools— to quantify this improvement in the portability and main- Autoconf [5], Automake [6], and Libtool [7]—addresses tainabilty of the Am-utils package, and those investiga- the aforementioned problems in five ways: tions led to writing this paper. There are six reasons why porting such packages to 1. Standard tests: Autoconf has a large set of stan- new platforms, adding new features, or fixing bugs be- dard portable tests that were developed from practi- comes a difficult task more suitable for automatic con- cal experiences of the maintainers of several GNU figuration: packages. Autoconf tests for features by actually exercising those features (e.g., compiling and run- 1. Operating system variability: There are more ning programs that use those features). Packages Unix systems available today, with more minor re- that use Autoconf tests are automatically portable leases, and with more patches. Flexible software to all of the platforms on which these tests work. packaging allows administrators to install selective 2. Consistent names: Autoconf produces uniform parts of the system, increasing variability. An auto- macro names that are based on features. For ex- mated build process can track small changes auto- ample, code which uses Autoconf can test if the matically, and can even account for local changes. system supports a reliable memcmp function us- 2. Code inclusion and exclusion: To handle ing #ifdef HAS MEMCMP, rather than depend- platform-specific features, large portions of code ing on system-specific macros (e.g., #ifndef are often surrounded by #ifdef directives. SUNOS4). Autoconf provides a single macro per Platform-specific code is mixed with more generic feature, reducing the need for complex or nested code. Often, system-specific source files are com- macro expressions. This improves code readability piled on every system, because there is no automatic and maintainability. way to compile them conditionally. 3. Shared libraries: By using Libtool and Automake 3. Multi-level nested macros: To detect certain along with Autoconf, a package can build shared or features reliably, older code uses deeply nested static libraries easily, removing a lot of custom code #ifdef directives. This results in complex macro from sources and makefiles. expressions designed to determine features as re- 4. Human factors: Building packages that use auto- liably as possible. The main problem with such tools is easy. Administrators are becoming increas- macros is that they provide second-hand or anec- ingly familiar with the process and the standard set dotal knowledge of the system. For example, to test of features autotools provide (i.e., run ./config- if a compiler supports “void *”, some code depends ure and then make). Administrators do not need 2 to configure the software package manually prior to #include "confdefs.h" compilation and they are likely to make fewer mis- char bzero(); /* forward definition */ takes. This standardization speeds up installation int main() and configuration of software. bzero(); return 0; 5. Extensibility: Finally, software maintainers can ex- ¡ tend Autoconf by writing more tests for specific needs. For example, we wrote specific tests for If the program compiles and links successfully, the Am-utils package that detect its interaction with the configure script defines a CPP macro named certain kernels. This allowed us to separate the HAVE BZERO in an automatically-generated configura- common code from the more difficult-to-maintain tion file named config.h. This macro is based on the platform-specific code. existence of the feature and can be used reliably in the sources for the package. Note that it is not necessary to Our experiences with maintaining the Amd package run this program to determine if bzero exists. In fact, clearly show the benefits of autotools. When we con- the above program will fail to run properly because the verted the Amd package [9, 12] to use autotools, the code bzero function was not given proper parameters. size was reduced by more than one-third and the code Assuming the package also checks for the existence became clearer and easier to maintain. Fixing bugs and of the memset function, the maintainers of the package adding new features became easier and faster, even ma- can use the following portable code snippet reliably: jor features that affected significant portions of the code: #ifdef HAVE_CONFIG_H NFSv3 [8] support, Autofs [1] support, and a run-time # include <config.h> automounter configuration file /etc/amd.conf.