Sendmail Evolution: 8.10 and Beyond Gregory Neil Shapiro [email protected] Eric Allman [email protected] Sendmail,Inc. 6603Shellmound Emeryville,California Originally presented at the 1999 USENIX Annual Technical Conference FREENIX Track Monterey, CA ABSTRACT SendmailTM has been the de facto mail transfer agent implementation since the dawn of the Internet. Today, sendmail development is still drivenbyacontinually changing set of network requirements and user demands. Lately,two new driving forces have also contributed to sendmail development. First, as more open source mail transfer agents, such as Exim and Postfix,become available, a newfriendly competition has developed in which the authors of the various MTAs share their ideas via open source and help to advance open standards as opposed to advancing their own particular implementation. Second, a new“hybrid” company, Sendmail, Inc., has been created to offer commercial versions of the open source software while continuing to fuel open source development. This paper will briefly discuss the evolution of sendmail;the influences which drive sendmail development; and howthe creation of Sendmail, Inc. has contributed to the open source version. The paper will also describe the newfeatures appearing in the next ‘‘functionality release’’ofopen source sendmail.Inparticular,changes in queueing and newprotocol support are discussed. Finally,the authors will speculate on future directions for sendmail. Copyright (c) 1999 Sendmail, Inc. All rights reserved. 1 1. Introduction shipped in 1983 with 4.1c BSD—one of the initial The sendmail mail transfer agent (MTA) is used operating systems to support TCP/IP. Sendmail on most UNIXTM systems today.Recent changes have accomplished twoimportant goals. First, it provided a influenced sendmail development, notably the creation reference implementation of the Network Working of a new“hybrid” companydedicated to supporting Group (later the Internet Engineering Task Force, or both the open source code as well as a commercial IETF) mail standard [Cost97]. Second, the version. configuration was read at run time to allow reconfiguration for different networks without Section 2 givesabrief history of sendmail. recompilation. Because of the wide variety of networks Section 3 describes the forces acting to influence supported, the configuration was designed to be changes in sendmail.Section 4 outlines Sendmail, friendly toward non-conforming addresses. Instead of Inc.’s effects on the open source. Section 5 discusses rejecting messages that were not acceptable to the changes appearing in sendmail 8.10.Future directions standard, it tried to repair them; this broad acceptance that sendmail may takeare laid out in section 6. of inputs maximized interoperability with other Finally,asummary and concluding remarks are networks available at the time, such as UUCP. presented in section 7. By late 1986, Allman’sinv olvement with sendmail had tapered off, and several other people picked up development. The most important version .. was IDAsendmail from Lennart Lovstrand of the .. University of Linkoping in Sweden, with later 2. History maintenance by Neil Rickert of Northern Illinois To understand the continuing evolution of University and Paul Pomes of the University of Illinois sendmail,you must first look at its history.Likemany [Cost97]. The most important feature added by IDA successful open source projects, sendmail started as a wasthe concept of external databases in DBM format. “scratch your itch” solution to a problem. Shortly thereafter,Paul Vixie, then at Digital Equipment, created KJS (King James Sendmail),an 2.1. In the Beginning... attempt to unify the divergent versions, but this version wasnot widely adopted. Sendmail had effectively Sendmail started out as delivermail,written by splintered. Eric Allman, then a graduate student and staffmember at the University of California at Berkeley. Delivermail 2.2. Sendmail 8Emerges solved the problem of routing mail between three different networks running on the Berkeleycampus at In late 1989, Allman returned to U.C. Berkeley, the time: the ARPAnet, UUCP,and BerkNet. The first and not long thereafter was drawn back into sendmail public version was distributed in 1979 as part of the development. By July of 1991, serious work on what Fourth BerkeleySoftware Distribution (4BSD) and would become sendmail 8 had begun. Manyideas were later as part of 4.1BSD [Allm85]. taken from IDAsendmail and KJS,although most were generalized. For example, external databases were Although delivermail solved the immediate added, but in such a way that formats other than DBM problem faced by Berkeley, itwas not generic enough were available. Sendmail 8.1 wasreleased with 4.4 to solvethe problems of other custom networks in BSD in mid-1993. Sendmail 8 quickly became a operation. Since the instructions for talking among the unifying influence, as vendors converted from their networks were part of the C source code, it was not hacked versions to the newer version. Some features easy for sites to reconfigure delivermail for their from vendor versions were also included in the new specific needs. The configuration was also not flexible release, for example, NIS support from Sun enough to handle complexmail environments. Microsystems. These additions are just one of many At the same time the ARPAnet was transitioning examples of the success of open source software: to the newInternet protocol, TCP/IP.Part of the new sendmail 8 wasfertilized with ideas from other open protocol suite included extracting mail transmission out source and vendor versions. of the file transfer protocol (FTP) into its own protocol, Another important change that occurred the Simple Mail Transport Protocol (SMTP) [RFC821]. concurrent with sendmail 8 wasthat versions were The user demand for a customizable program and controlled more carefully.The previous major release the network requirements created by the newmail (sendmail 5)had no fewer than 143 “dot” releases (that protocol led to the creation of sendmail,which first is, 5.1 through 5.143), often more than one in a single Copyright (c) 1999 Sendmail, Inc. All rights reserved. 2 day.Some of those were intended for public years. Unfortunately,with the growth of spam on the consumption, some were test releases. With version 8, Internet, this default is no longer acceptable. sendmail switched to a policyofclearly labeling test The increasing use of email as a vector of viruses releases, producing production releases less often, and has heightened the need for MTAs to include content clearly identifying newfunctionality releases from bug- checking. An SMTP server running on a firewall must fix releases. This change in release frequencywas be prepared to vet the data it is handling. Because of essential to the wide acceptance of sendmail 8 by the this need, 8.9 included message header checking and community.The downside of this change is that people 8.10 will include a mail filter API for more advanced who liketobeonthe “bleeding edge” have towait header and body filtering. longer,and newfeatures are not tested immediately. We viewthis loss of quick feedback as being an Changes in Internet standards from the IETF also acceptable tradeoff. have a major impact on sendmail.During 1998, the IETF accepted 22 newRFCs that involved electronic An unfortunate effect of the success of sendmail mail in one form or another.Atleast one of these 8 wasthat Allman quickly became overloaded with RFCs has a direct effect on MTAs such as sendmail. answering questions. This overload was the impetus RFC 2476, Message Submission, specifies a separate behind the establishment of the Sendmail Consortium, protocol for initial insertion of a newmessage into the aloosely-knit group of volunteers providing free message delivery system using SMTP [RFC2476]. support for sendmail.Gregory Shapiro was invited to join that group during the 8.8 cycle, and by 8.8.6 was As of April 1999, three more MTA-related RFCs doing a large part of development and most of the have already appeared. RFCs 2487 [RFC2487] and release engineering, although Allman continued to 2554 [RFC2554] provide encryption and authentication reviewand approve changes. for an SMTP session. RFC 2505 [RFC2505] is a set of recommendations for features that MTAs should In 1997, Allman found that evenwith the help of provide to combat spam. Additionally,the Detailed an extremely capable volunteer staff, he was unable to Revision/Update of Message Standards (drums) IETF keep up with the support load and continue to move Working Group is preparing to release the long-awaited sendmail forward. After exploring several other update to RFCs 821 and 822, probably later in 1999 approaches for adding resources for sendmail [SMTPUPD, MSGFMT]. Clearly,the messaging development, he finally settled on founding a “hybrid” standards landscape is not static. business model companytoproduce a commercial version of sendmail while continuing to support and 3.2. User Demands extend the open source version. By using the “hybrid” approach, he was able to protect the interests of the By far,howev er, most feature requests come open source community while creating a viable from sendmail users. It is common for the Sendmail business model. Consortium to receive three to fivefeature requests per week, some complete with the patches necessary to implement the feature. These feature requests produced a list of 320 requests before 8.10 development evenbeg an. When deciding which features to implement and 3. Driving Forces