2004 USENIX Annual Technical Conference
USENIX Association
Proceedings of the FREENIX Track: 2004 USENIX Annual Technical Conference
Boston, MA, USA June 27–July 2, 2004
© 2004 by The USENIX Association All Rights Reserved For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. The NetBSD Update System
Alistair Crooks, The NetBSD Project 9th April 2004
Abstract driving force behind the use of pkgsrc or NetBSD - rather, this is a description of a facility which is This paper explains the needs for a binary patch and used in NetBSD and which can be used on any other update system, and explains the background and im- operating system to augment the standard facilities plementation of NetBSD-update, a binary update fa- which are in place. cility for NetBSD. The implementation is then anal- ysed, and some lessons drawn for others who may be interested in implementing their own binary up- Driving Forces for a Binary Patch and date system using the NetBSD pkgsrc tools, which Update System are available for many operating systems and envi- ronments already. It is now common to nd rewalls in large and small organisations, preventing malign access, and protect- ing the organisation from intrusion and other attacks. The NetBSD Binary Update Sys- It would not be prudent to have a C compiler in- tem stalled on such a machine - its use should be that of a gatekeeper, as a bouncer with an attitude, keep- Unix, Linux and the BSD operating systems have ing anything suspicious out, and not allowing anyone traditionally been distributed in source format, and who does manage to penetrate the defences to use users and administrators have had a long tradition any tools to break further into the infrastructure. of compiling utilities and applications from source. In addition to these instances, it is very unusual to Over time, however, vendors have moved towards a nd users (as opposed to administrators, or people binary-only distribution mechanism, removing vari- who work in the industry) who would know what ous parts of the system in the process, such as the C to do with a source distribution. Email to various compiler, and other necessary tools. It is only over mailing lists proves this point - to the majority of the last decade that the rise of Linux and the BSD users out there, the computer is a tool, not a thing operating systems have placed the emphasis back on of beauty. source code, and even then, most versions of the op- It is also common to nd vulnerabilities in oper- erating systems are installed from a binary distribu- ating systems, libraries, and utilities which have al- tion. ready been deployed. To ll such holes, a patching This paper describes the NetBSD update sys- system needs to be used - the vulnerable code is re- tem, which has been implemented on and for the placed by code which is not vulnerable. This must NetBSD operating system http://www.netbsd.org/ be done by means of updating the binaries. using the packaging tools from the NetBSD Packages Embedded systems must also be examined in light Collection (pkgsrc) http://www.pkgsrc.org/. of the vulnerabilities found in MTAs, network time Other vendors and operating systems have binary servers, IP ltering software, operating system soft- update facilities in place - their existence is not the ware, and anything else which is included as part of
1 the embedded system, and which can be used to facil- The RedHat network http://www.redhat.com/ itate an attack on an organisation or individual - the provides a commercial service for RedHat Linux sys- whole infrastructure is only as strong as the weakest tems. link in the defences, and one breach renders the rest Sun Microsystems http://www.sun.com/ provides of those defences useless. a number of binary updates via its SunSolve facility It is not feasible to expect that all the vulnera- for its Solaris operating environment. bilities can be found and protected ahead of time - The NetBSD Packages Collection http://www. new ones are being found every day - and so there pkgsrc.org/ has an audit-packages package which must be a mechanism available by which new bina- is used to notify the user when packages which are ries can be used to overwrite vulnerable ones. This installed on the host system have been found to have paper examines all the issues involved in the design security exploits or vulnerabilities. A central list and deployment of such a system. is maintained by the security-o cer and other de- It can take some time for an advisory to be re- velopers, and stored on one of the NetBSD project ceived, investigated, researched, xed, and then pub- servers. The user is encouraged to run the download- licised - the binary update facility can be viewed as a vulnerability-list as part of the periodical jobs on more reliable form of communication of the exploit or a host system by means of the crontab(8) facility. vulnerability than reading the Bugtraq or Full Dis- This script will download the list of vulnerabilities closure mailing lists. from the central server. Output from the download- Some vendors wish users and administrators to pay vulnerability-list script looks is reproduced in the for the added service which a binary update facility Figure download-vulnerability- le output. provides. Whilst the NetBSD binary update system A separate script, called audit-packages simply is not intended to prevent vendors charging for this runs through each of the entries in the list of vulnera- service, some people are unhappy and unwilling to bilities, and checks against each of the vulnerabilities pay for a binary update system. for an installed package on the system with a version number that is vulnerable. If there is, then a warning Related Systems message will be printed, with relevant information, as shown in Figure audit-packages. The one immediate piece of software to which The vulnerabilities le is split into 3 columns. Each everyone refers when talking about binary up- line of the le describes a new vulnerability. Com- dates is the Windows Update Facility http://v4. ments start with the '#' character. The rst column windowsupdate.microsoft.com/en/default.asp. is the vulnerable package versions; the second column It has three separate facilities - update check (for is the type of vulnerability, and the nal column is a new upgrades), update download, and update instal- URL related to the vulnerability. lation, and Windows XP computers can be set to Some typical entries from the vulnerabilities list perform the check, the check and download, and all are shown in the Figure pkg-vulnerabilities le. three parts automatically. Windows Update is based The audit-packages package works very eectively around a web front end - all the used has to do is to in practice, having ironed out a few loose ends in its specify (the rst time Windows Update is run) how implementation. From the start, the vulnerability list they would like their system to check for updates - has been advertised that it will only grow - old vul- and everything is very easy from an end-user point nerabilities will be retained just in case. This has of view. proved di cult to implement - occasionally URLs will Debian Linux http://security.debian.org/ has be found to be out of date, or better ones identi ed. a security update feature which uses its apt system Once a user with a shorter username xed two trans- to provide binary patches for Debian Linux systems. posed characters in a comment, thereby causing the The FreeBSD project http://www.freebsd. size of the vulnerabilities list to decrease. In time, org/are working on their own binary update system. it was found better to have an embedded SHA1 di-
2 gest, which is used to ascertain correct tranmission The author has some experience with the pack- of the vulnerabilities list, and that no truncation has aging tools as used in the NetBSD packages collec- occurred in transit, as well as ensuring that the le tion, and certain facilities used in these tools would has not been modi ed in any way. be needed in the binary update facility itself. Using NetBSD also has ne-grained system packages, version numbers which could be compared rationally which can be used as an alternative to the traditional with each other, and using the tools themselves to BSD sets method of installing or upgrading systems. add, backup and delete les, directories and other System packages are usually made up of three sub- le system entries are essential in providing a useful packages: utility which does not try to do any more than it needs to do. • the binaries, Implementation of the NetBSD Update the manual pages, • System and any example or support les. • The implementation of the NetBSD update system is done as in the Windows and Debian updates - the The NetBSD Update System does not use any of the binary updates can be set to inform of new updates, facilities of system packages - on non-NetBSD sys- to inform and download, and to inform, download tems, it is likely that system packages would clash and install the updates automatically. with traditional methods of installing and updating The NetBSD packages collection tools are used systems, and so a separate mechanism was used to to perform the binary update work - these tools implement the NetBSD Update System. have been enhanced from the original tools to add such features as Dewey decimal number compar- The Design of the NetBSD Update Sys- ison, digitally-signed package checks and addition, tem and package replacement in place. There is also a facility for a binary package to be speci ed by The audit-packages package, and the experience URL on the command line, downloaded automati- gained from it, proved to be a major in uence in the cally, and added via pkg_add(1), but that facility is design of the NetBSD update system. By using a sin- not used by the NetBSD binary update system - in- gle list of vulnerable packages which was downloaded, stead, ftp(1) isused to download the binary package, signi cant useful information could be retrieved very and pkg_add(1) is used in local mode. This is be- simply and in a relatively secure manner. cause we prefer the binary packages which are being The utility and ease-of-use of the Windows Update downloaded to be protected by a digital signature, system proved to be the inspiration for the actual and that mode of binary package addition is, unfor- operation of the system. In addition to the ease of tunately, not yet available when specifying a URL on use, the three stages of protection in the Microsoft the command line. utility were considered to be the correct approach. For a long time, the NetBSD pkgsrc system has in- In some countries, users must opt-in to any service cluded a package called pkg_install - it consists of which holds information, and the binary update fa- the latest version of the pkg_install tools. This was cility was designed with this in mind. derived from the original FreeBSD pkg_install(1) Security and integrity were two more considera- tools (and is also included in the base NetBSD oper- tions - security in the transit of information which ating system), but has been considerably enhanced, could be potentially damaging to a system if the up- including speci cally such such features as date were modi ed in any way during transit. For this reason, digital signatures of the update itself are • the recognition of packages by means of relative considered absolutely essential. version numbers
3 • the addition of a callout to gpg and pgp to au- • gnome thenticate the provenance of a binary package by means of a digital signature related to the bi- • gnome2 nary package. This had to be implemented as • gnustep a callout as gpg is distributed under the GPL, and it is not desirable to have any part of this • kde3 system distributed under a more restrictive or onerous licence than the BSD licence. The im- • netbsd-www plication of this is obvious - the BSD operating etc systems need a BSD-licensed utility for privacy, • compatible with gpg and pgp. One example of an addition of a signed binary package is shown The First Implementation in Figure Addition of a signed binary package. As it was rst implemented, the NetBSD binary up- The recognition of packages by comparitive ver- date system was actually a fairly simple shell script sion numbers is essential for any binary update sys- which used o-the-shelf tools to perform its work. tem. For example, it is imperative that we can tell Some thought had been placed into making it into a whether an installed package is vulnerable to an ex- standalone compiled program, since that is the vehi- ploit. There are a number of ways this could be done cle which would most readily bene t embedded sys- tems vendors, and that is left as future work. • circulate digests of les that are vulnerable, but The system itself consisted of two parts: that is not practical in an environment such as the BSD operating systems where most packages • an $opsys-$osversion-$architecture-update are built by the user, and binary packages, whilst meta-package still being used by many people, are not as pop- ular as the build it for myself attitude. The • the individual updates in the form of binary author suers from this a iction as well, and packages does not consider it a weakness, more the signs of knowing exactly what is on any given system The netbsd-update facility rst checked the directory at any one time. on the NetBSD ftp server to see if there was a more recent meta-package than the one currently installed • by hardcoding the package names which are vul- on the host machine. If there was, this was noti ed nerable, but this does not scale well, or would to the user. even be practical Then, when the time came to download the new updates, the new meta-package was downloaded, as • by using a system package name and version well as any binary packages which were necessary. number (but, as we have already seen, this is Whilst it was usual for an update to include one bi- not portable to operating systems other than nary package for that architecture and version, more NetBSD) may be needed. It's also possible that a binary pack- The NetBSD packages collection has also long had a age may itself have been updated, in which case the number of meta-packages - these are packages which newer version of that was downloaded, along with the include no les or directories of their own, but which updated meta-package. require pre-requisites of other packages, and thereby The operating system version and architecture are ensure that all of the parts can be manipulated as used in the package name to ensure that the correct one whole package. Examples of meta-packages are: binaries are downloaded (the architecture), and that the binary package was linked with the correct system • edesktop libraries (operating system version).
4 When the time came to install the binary updates, packages, and made into a binary package of its the normal pkg_add(1) tool was used, working from own. This also facilitates rollback, should an un- a local copy of the package. The -s switch was used successful binary package addition take place, or to ensure that the digital signature (in the form of a the new behaviour with the update in place is separate ASCII-armoured le) for the binary package not working as intended veri ed the update's authenticity (much as a signed 8. the binary package is fed to the pkg_add(1) RPM package is used in Linux). utility, which rst checks the digital signature on the binary package and veri eds it It is An overview of the NetBSD Update Fa- possible to specify that any signatories which cility match a regular expression will be allowed - for example security-o cer@netbsd\.org, or Building on the prototype building blocks outlined .*@pkgsrc\.org above, the NetBSD Update facility has to perform the following steps: 9. the binary package is added to the system by means of pkg_add(1). 1. ascertains the current operating system name and version Consequences of these steps:
2. downloads a list of the binary packages available • the user is able to specify in advance whether for the operating system and version of the op- they want the binary package noti cation, auto- erating system matic download, or applied to the system. This is the same system as the Windows Update facil- 3. evaluate whether any binary packages are needed ity, and has been found by numerous people from - the same process which is undertaken by audit- small users to large enterprises to work very well. packages takes place here. The list of available binary packages is processed, using pkg_info(1) • digital signatures protect the user by assuring to work out whether the binary package is in- them of the provenance of the binary package. stalled on the system or if it needs to be. Without this assurance, it would be risky or downright dangerous to apply third-party binary 4. if the user has speci ed that they just want to packages to any system, far less one which may be informed of the existence of an update (which be in production use. is not recommended), then this takes place by means of mail to root, and no further actions • the ability to rollback from an update is simply take place. the deletion of the downloaded binary package, and the re-application of the preserved binary 5. the necessary available packages are downloaded package. from the ftp server, along with their digital sig- people must be running a GENERIC kernel for natures. • the update system to be of any use. This is not 6. if the user has speci ed that they just want the really a problem in reality, since very few peo- available binary packages downloaded, then they ple tailor or customise their kernel con gurations are informed by mail (to root) that this has hap- (usually, the only people that do that are devel- pened, and no further actions take place opers)
7. the les which will be overwritten (if any) by ex- Feedback from Initial Usage tracting les from the available binary packages are preserved by means of a simple tar(1) com- • The user interface was originally clunky, and mand, using the le list from the available binary needed to be made smoother
5 • Problems with Firewall-1 and interaction with • a digital signature of the vulnerabilities le can NetBSD's ftp server mean that http may well be be added in a trivial manner, to protect the in- a better vehicle for binary update downloading tegrity of the information
• The digital signature and automatic updates do • using a simple vulnerabilities le means that not co-exist very well together. It is anticipated there is no confusion of dierent les per archi- that the pkg_install tools will be modi ed to use tecture or operating system, no duplication of a con guration le for trusted signatures. information
• Operating systems which are not NetBSD were The re ned binary update facility now performs the not catered for very well. Having said that, there following steps: was nothing to preclude their use on other oper- ating systems, although duplication of updates • download the vulnerabilities le from the from Sunsolve, for example, are not useful in NetBSD ftp server themselves; what is useful, however, is a coher- ent set of packages for a given release of Solaris, • ensure its integrity and provenance by calling out for relevant architectures and machines. It has to gpg to verify the digital signature been the author's experience in the past, when managing networks of Solaris machines, that Sun • if there is a vulnerability on the host system, this provide the updates, but do not group them to- will be agged by a down-level update package gether particularly well, and that the numerical (from a previous vulnerability x), or by the up- nature of the updates names are not the easiest date package not existing on the host machine. to remember or inform others. • if the update package is to be downloaded, use • The quality of the binary updates is only as good ftp(1) to download the binary package from the as the port-masters for NetBSD who have to NetBSD ftp server. At the same time, the digital build the binary packages. signature will be downloaded. Typically, they will both be downloaded together as one entity, • Making meta packages for each individual archi- to try to minimise any spoo ng attacks, or man tecture is onerous and time-consuming. Some in the middle attacks, since ftp is hardly the most knowledge of package creation is needed. secure of protocols
Iterative Development • if the update package is to be installed automat- ically, if there are any les to be preserved (so Learning from the experience gained from the pro- that rollback can take place), then the package totype implementation, it was decided that some tools are used to create a binary package of the changes would bene t everyone: preserved les. The list of les to be preserved will be included with the update package itself • rather than having a separate meta-package for as part of its meta data. each operating system, and operating system version, and architecture, a single vulnerabilities • if the update package is to be installed, use le was used, mirroring the vulnerabilities le in pkg_add(1) with its -s gpg argument to ver- the audit-packages package ify the package's integrity, and to add it
• the vulnerabilities le can also use an embedded • the root user will be informed of all of the above digest, as a further check that correct transmis- steps by email. It is believed by the author that sion has taken place - although this is not as it is better to err on the side of too much infor- useful or protective as: mation, especially where far-reaching changes to
6 the system could be taking place. Update pack- • OpenBSD ages can have their information displayed using the normal pkg_info(1) interface. • SunOS (Solaris) whilst more - HP/UX, the Hurd, Digital Unix - are Conclusions planned. We would like to extend the binary update The binary update system is a useful piece of work, facility to run on as many of those operating sys- which has shown positive bene ts. tems as possible, as the bene ts are measurable. It is certainly the case that duplication of work would • The ability to provide better security warnings be inadvisable, but the binary update facility oers and recovery is an enormous bene t a real bene t to end users. It would be bene cial to investigate using a pro- • even if a x is not available, the binary update tocol that is more secure than ftp to download the system can be used to publicise an exploit in vulnerabilities le, update packages, and digital sig- the wild, thereby making the administrator's job natures. Some form of anonymous secsh could be easier (assuming that information about the ex- useful for this project. ploit is available). Even being aware that a vul- A BSD-licensed gpg utility, perhaps based on nerability has been found in a widely-used piece openssl, would mean that callouts to gpg could be of software is a valuable service avoided, and that the whole binary update system would be licensed with a less restrictive licence than the use of digital signatures to protect and dis- • the GPL. seminate the information has proved to be of im- mense bene t References There are certainly areas for development and improvement, but the cross-platform and cross- 1. The NetBSD Operating System - environment nature of the NetBSD packages collec- http://www.netbsd.org/ tion can aid other operating systems and environ- 2. The NetBSD Packages Collection - ments as well as simply the BSD operating systems. http://www.pkgsrc.org/
Future Work 3. The Microsoft Windows Update Facility - http://v4.windowsupdate.microsoft.com/en/default.asp The NetBSD packages collection (pkgsrc) runs on the following operating systems: 4. Debian Linux Security Update - http://security.debian.org/ • AIX 5. The FreeBSD Project - http://www.freebsd.org/ • BSDOS 6. The Red Hat Network - • Darwin (Mac OS X) http://www.redhat.com/ • FreeBSD 7. Sun's SunSolve database - http://sunsolve.sun.com/ • Interix (Microsoft's Services for Unix) • IRIX • Linux • NetBSD
7 Figure 1: download-vulnerability- le output
8 Figure 2: audit-packages
9 Figure 3: pkg-vulnerabilities le
10 Figure 4: Addition of a signed binary package
11