THE ADVANCED COMPUTING SYSTEMS ASSOCIATION The following paper was originally published in the Proceedings of the FREENIX Track: 1999 USENIX Annual Technical Conference Monterey, California, USA, June 6–11, 1999 Sendmail Evolution: 8.10 and Beyond Gregory Neil Shapiro and Eric Allman Sendmail, Inc. © 1999 by The USENIX Association All Rights Reserved Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org Sendmail Evolution: 8.10 and Beyond Gregory Neil Shapiro [email protected] Eric Allman [email protected] Sendmail, Inc. 6603 Shellmound Street Emeryville,California 94608 ABSTRACT SendmailTM has been the de facto mail transfer agent implementation since the dawn of the Internet. Today, sendmail development is still drivenbyacontinually changing set of network requirements and user demands. Lately,two new driving forces have also contributed to sendmail development. First, as more open source mail transfer agents, such as Exim and Postfix,become available, a newfriendly competition has developed in which the authors of the various MTAs share their ideas via open source and help to advance open standards as opposed to advancing their own particular implementation. Second, a new“hybrid” company, Sendmail, Inc., has been created to offer commercial versions of the open source software while continuing to fuel open source development. This paper will briefly discuss the evolution of sendmail;the influences which drive sendmail development; and howthe creation of Sendmail, Inc. has contributed to the open source version. The paper will also describe the newfeatures appearing in the next ‘‘functionality release’’ofopen source sendmail.Inparticular,changes in queueing and newprotocol support are discussed. Finally,the authors will speculate on future directions for sendmail. 1. Introduction 2. History The sendmail mail transfer agent (MTA) is used To understand the continuing evolution of on most UNIXTM systems today.Recent changes have sendmail,you must first look at its history.Likemany influenced sendmail development, notably the creation successful open source projects, sendmail started as a of a new“hybrid” companydedicated to supporting “scratch your itch” solution to a problem. both the open source code as well as a commercial version. 2.1. In the Beginning... Section 2 givesabrief history of sendmail. Sendmail started out as delivermail,written by Section 3 describes the forces acting to influence Eric Allman, then a graduate student and staffmember changes in sendmail.Section 4 outlines Sendmail, at the University of California at Berkeley. Delivermail Inc.’s effects on the open source. Section 5 discusses solved the problem of routing mail between three changes appearing in sendmail 8.10.Future directions different networks running on the Berkeleycampus at that sendmail may takeare laid out in section 6. the time: the ARPAnet, UUCP,and BerkNet. The first Finally,asummary and concluding remarks are public version was distributed in 1979 as part of the presented in section 7. Fourth BerkeleySoftware Distribution (4BSD) and later as part of 4.1BSD [Allm85]. Although delivermail solved the immediate added, but in such a way that formats other than DBM problem faced by Berkeley, itwas not generic enough were available. Sendmail 8.1 wasreleased with 4.4 to solvethe problems of other custom networks in BSD in mid-1993. Sendmail 8 quickly became a operation. Since the instructions for talking among the unifying influence, as vendors converted from their networks were part of the C source code, it was not hacked versions to the newer version. Some features easy for sites to reconfigure delivermail for their from vendor versions were also included in the new specific needs. The configuration was also not flexible release, for example, NIS support from Sun enough to handle complexmail environments. Microsystems. These additions are just one of many At the same time the ARPAnet was transitioning examples of the success of open source software: to the newInternet protocol, TCP/IP.Part of the new sendmail 8 wasfertilized with ideas from other open protocol suite included extracting mail transmission out source and vendor versions. of the file transfer protocol (FTP) into its own protocol, Another important change that occurred the Simple Mail Transport Protocol (SMTP) [RFC821]. concurrent with sendmail 8 wasthat versions were The user demand for a customizable program and controlled more carefully.The previous major release the network requirements created by the newmail (sendmail 5)had no fewer than 143 “dot” releases (that protocol led to the creation of sendmail,which first is, 5.1 through 5.143), often more than one in a single shipped in 1983 with 4.1c BSD—one of the initial day.Some of those were intended for public operating systems to support TCP/IP. Sendmail consumption, some were test releases. With version 8, accomplished twoimportant goals. First, it provided a sendmail switched to a policyofclearly labeling test reference implementation of the Network Working releases, producing production releases less often, and Group (later the Internet Engineering Task Force, or clearly identifying newfunctionality releases from bug- IETF) mail standard [Cost97]. Second, the fix releases. This change in release frequencywas configuration was read at run time to allow essential to the wide acceptance of sendmail 8 by the reconfiguration for different networks without community.The downside of this change is that people recompilation. Because of the wide variety of networks who liketobeonthe “bleeding edge” have towait supported, the configuration was designed to be longer,and newfeatures are not tested immediately. friendly toward non-conforming addresses. Instead of We viewthis loss of quick feedback as being an rejecting messages that were not acceptable to the acceptable tradeoff. standard, it tried to repair them; this broad acceptance An unfortunate effect of the success of sendmail of inputs maximized interoperability with other 8 wasthat Allman quickly became overloaded with networks available at the time, such as UUCP. answering questions. This overload was the impetus By late 1986, Allman’sinv olvement with behind the establishment of the Sendmail Consortium, sendmail had tapered off, and several other people aloosely-knit group of volunteers providing free picked up development. The most important version support for sendmail.Gregory Shapiro was invited to .. was IDAsendmail from Lennart Lovstrand of the join that group during the 8.8 cycle, and by 8.8.6 was .. University of Linkoping in Sweden, with later doing a large part of development and most of the maintenance by Neil Rickert of Northern Illinois release engineering, although Allman continued to University and Paul Pomes of the University of Illinois reviewand approve changes. [Cost97]. The most important feature added by IDA In 1997, Allman found that evenwith the help of wasthe concept of external databases in DBM format. an extremely capable volunteer staff, he was unable to Shortly thereafter,Paul Vixie, then at Digital keep up with the support load and continue to move Equipment, created KJS (King James Sendmail),an sendmail forward. After exploring several other attempt to unify the divergent versions, but this version approaches for adding resources for sendmail wasnot widely adopted. Sendmail had effectively development, he finally settled on founding a “hybrid” splintered. business model companytoproduce a commercial version of sendmail while continuing to support and 2.2. Sendmail 8Emerges extend the open source version. By using the “hybrid” In late 1989, Allman returned to U.C. Berkeley, approach, he was able to protect the interests of the and not long thereafter was drawn back into sendmail open source community while creating a viable development. By July of 1991, serious work on what business model. would become sendmail 8 had begun. Manyideas were taken from IDAsendmail and KJS,although most were generalized. For example, external databases were 3. Driving Forces When deciding which features to implement and As can be seen in the preceding section, sendmail howtheyshould be implemented, we try to balance has responded to both changing network requirements backwards compatibility with change. By introducing and user demands. In addition to these demands, new radical changes gradually,wegiv e sendmail sites a open source MTAalternativeshelp in driving sendmail chance to prepare for the changes. Acombination of a forward. huge user population and 20 years of sendmail availability prevents us from doing radical changes 3.1. Network Requirements without advanced warning. For example, the 8.9 documentation included a notice warning users that The network requirements come both from the configuration file names would be changing in 8.10. changing face of the Internet and from newInternet Also in 8.10, the LDAP map class will be changed drafts and RFCs from the IETF.For example, up until from ldapx to ldap,thereby dropping the class version 8.9, sendmail allowed third party,promiscuous name’sconnotation as an experimental map. The old relaying by default. This willingness to relay had been name will continue to work (and print a warning) in an acceptable, evendesirable, default for more than 15 8.10, but will be removedina subsequent release. years. Unfortunately,with the growth of spam on the Some of the other open source mail transfer agents, Internet, this default is no longer acceptable. such as Postfix and qmail,are not yet so constrained. The increasing use of email as a vector of viruses has heightened the need for MTAs to include content 3.3. AlternativestoSendmail checking. An SMTP server running on a firewall must At the same time, these other open source MTAs be prepared to vet the data it is handling. Because of also drive sendmail development.