Qmail in the FreeBSD Universe
Renato Botelho, Erwin Hoffmann
EuroBSDCon Karlsruhe/Germany October 2010
1 Introduction Qmail+Spamcontrol with SPF to provide an even more complete solution for FreeBSD Qmail is a Mail Transfer Agent MTA submit- available as a port. ted by Daniel J. Bernstein as open source in This paper tries to outline some of basic 1998 in the still (final) version 1.03. Since then, ingredients of the design concepts and usage only very minor bugs have been identified (but under FreeBSD. never corrected for by the author), while the (E)SMTP protocol has significantly changed 1.2 Authors and enhanced. However, the superior design of Qmail and in Renato Botelho is using FreeBSD since 1999. particular the new unrestricted distribution li- In 2004 he started to contribute to Ports Col- cense allows a smart adoption to the current lection and in addition setting up Qmail as needs, which reflects its authority to be still a MTA in conjunction with Spamcontrol, as be- major MTA. ing described in some Brazilian tutorials. In march 2005, he added the Spamcontrol slave 1.1 Scope port, became shortly after that the maintainer of all Qmail ports and finally became FreeBSD A lot of attemps have been made, to add ports committer in July the very same year. additional functionality to Qmail 1.03. Russel Right now he is working at BluePex Security Nelson’s Web site www.qmail.org represents Solutions in Limeira/Brazil dealing with Linux a good sample of enhancements. However, and FreeBSD based Security Appliances. the diversities of the third-party develop- Erwin Hoffmann uses Qmail since version ments, with different scopes, ’philosophies’, 0.99 and FreeBSD since 3.2 back in 1998. He is and maturity level makes it difficult and originator of the Spamcontrol patch for Qmail frustrating for a potential Qmail user to adopt and other Qmail-related tools. Currently, he the required ’patches’ and merge them into a is Associate Professor at the University of Ap- functional Qmail setup. plied Sciences in Frankfurt/Main Germany. Erwin Hoffmann’s Spamcontrol is an attempt to accommodate Qmail to the requirements of the 21st century. Unlike heavy sets of patches 2 Design and Components (e.g. Bill Shupps ’Qmail Toaster’), it provides 2.1 Components some lean improvements which have beeen incorporated by many other developers. Since Dan Berstein has published Qmail in the Renato Botehlo has enhanced mid of the 1990’s he has successfully raised
1 a lot of companion products which improve • ucspi-tcp is a set of services to be used Qmail as a MTS substantially: in conjunction with qmail-smtpd and qmail-qmtpd since both rely on network • During those days Dan Berstein wrote socket-services, which are otherise com- Qmail, some other (SMTP) MTAs were monly known as inetd and xinetd: typically used. Certainly, sendmail was the major favorite, though it was recog- – tcpserver is the main working horse, nized to have a substantial history of bugs. which typically is used as tcpwrapper. smail and zmailer were other candidates. – rblsmtpd provides a flexible lookup Postfix and Exim were at their dawn in into Relay Black Lists (RBL). those early day. Qmail included some revolutionary con- • ucspi-ssl is the alternative to ucspi-tcp cepts: and written by William E. Baxter from Superscript5. In particular sslserver – QMTP as Quick Mail Protocol. complements tcpserver’s capability mar- – VERP the Variable Address Re- rying it with the OpenSSL library for sponse Path, now cannibalized as transport layer encryption TLS. BATV1. • djbdns has revolutionized the recognition – Maildir, store one email as file and of the DNS protocol. Before the area of not stacked into a mbox. djbdns, BIND was considered simply as – Isolating MTA2 from the MDA3 and a synonym for DNS services. the MS4 processed with minimal per- missions. – dnscache is a DNS full-resolver and cache server, though without – Virtual Domains: Domains glued to DNSSEC capabilities. a Unix user instead of the MDA de- fault user. – tinydns is a DNS content server, which avoids the complexity of a • Daemontools is a set programs to care a BIND zone file in a ingenious con- about daemon programs utilizing cept.
– supervise, which not only provides Furthermore, Dan Bernstein introduced high availability HA for those dae- initially the concept of DNS port ran- mons but in addition with domization which became famous by Dan – multilog the logging is straightened Kamininsky’s description of the DNS and a unique TAI64 timestamp is birthday attack 6. However, this was al- written to the log output. ready recognized (and solved) by djbdns years before. Disentangling the tasks of a 1 Bounce Address Tag Validation draft by John DNS content server from those of a DNS Levine 2MTA: Mail Transfer Agent, responsible to transfer resolver and introducing a so-called split email from hop to hop via (E)SMTP horizon view while responding to DNS 3MDA: Mail Delivery Agent, receives the email and queries can additionally be attributed to stores in into a private container) 4MS: Message Store: Email is a store-and-forward 5www.superscript.com system, the MS is effectively the ’Queue’ 6https://www.kb.cert.org/vuls/id/800113
2 Dan Bernstein. Nobody even thinks today Product #Files #WoC about running an open email relay. The Exim 4.66 272 1.125.411 very same approach is valid now regarding Postfix 2.7.1 792 1.105.774 DNS caching services. Qmail 1.03 279 180.553 + Spamcontrol 2.6.23 323 231.003 2.2 Qmail + ucspi-tcp 509 298.938 + ucspi-ssl 711 389.273 Qmail 1.03 unlike – sendmail – is a highly composite piece of software. With it’s basic Table 1: Comparision of the code base (Words design, it includes of Code) for Exim, Postfix, and Qmail • a (E)SMTP client (qmail-remote) and server (qmail-smtpd), and his approach for simplicity. Most of those developments are also separately available as • a QMTP client (qmail-qmqpc) and add-ons for vanilla Qmail8. server (qmail-qmqtd, qmail-qmqpd), As already mentioned, one of the fundamen- • a POP3 server (qmail-pop3d), tals of Qmail and Spamcontrol is it’s minimal- istic approach and avoiding initial complex so- • a local Mail Delivery Agent (qmail- lutions and of course coding. local), and – of course – This becomes obvious, if we compare Qmail+Spamcontrol with the current available • a Message Store (qmail-queue). competitors Philip Hazel’s Exim9 and Wietse 10 Compared to other MTAs, Qmail does not pro- Venema’s Postfix [Table 1]. vide enough contemporary services, whether Though this count disfavors Qmail substan- ESMTP RFC-compliant, nor best-practice tially, we can see that the executable code base (like Greylisting). However, it’s design and it’s for Qmail is roughly 1/3 of the other MTAs. robustness makes it an ideal platform for fur- Conceptually, Qmail stays light-weight in the ther enhancements. For example, IndiMail7 is system and should perform as such. a current fork of Qmail. 3.1
3 Spamcontrol One criticism, in particular carried out by Matthias Andree11, is due to the fact, Spamcontrol is a patch for vanilla Qmail which that Qmail accepts any email in the first was started in the year 2000, mainly to in- place which targets to an acceptable domain clude some already existing patches for Qmail (control/rcpthosts). Since today spam and to adjust Qmail in particular to the needs mails simply use forged recipient addresses, of public german providers while incorporat- the generated Bounce email (NDR) is a heavy ing some anti-spam features in particular for load on the system. In particular, because the qmail-smtpd. advertised
3 always fake, this results in unwanted double- to the lack of an accepted end-to-end encryp- bounces. tion, Transport Layer Sucrity TLS is a reason- Here, Spamcontrol provides with the Re- able mean to provide confidentially at least to cipient mechanism a state-of-the art solution. the next (E)SMTP relay, and perhaps the final qmail-smtpd may ask a particular back-end destination. for address-validation. The back-end might be Based upon the ucspi-ssl environment, a local cdb (Constant Data Base) or (currently Qmail+Spamcontrol provides the means supported only) a LDAP directory. Since the for TLS both while receiving (qmail- Recipient extension facilitates Dan Bernstein’s smtpd/qmail-pop3d) and sending (qmail- checkpassword API by means of a PAM, remote). However, the complexity and (often practically any remote source can be queried. recognised) incapability of today’s PKI Public Key Infrastructure is not yet exploited fully in 3.2 ESMTP Authentication Spamcontrol regarding CRLs Certificate Re- vocation Lists and OCSP Online Certificate While SMTP was originally designed as Host- Status Protocol. to-Host protocol, with ESMTP Authentication Here, Spamcontrol supports fully ’virtual it is actually acting as a User-to-Host commu- hosting’ for TLS: nication. Once authorized (typically by means of Userid and Shared Secret), the MTA enables • Separately hosted domains can be bound for the authenticated user the privilege for un- to specific qmail-smtpd instances on dif- restricted relaying. If this information is trans- ferent IP addresses presenting a domain- mitted over un-encrypted channels, the Userid specific X.509 certificate instead the stan- is subject of a substantial data leakage, while dard host certificate. at least by means of the Challenge/Response method CRAM-MD5, the Shared Secret can • Reversely, qmail-remote can be ad- be protected. viced to bind to individual IP addresses Apart from the different authentication based on the
4 3.4 Virus prevention • A patched version of rblsmtpd permits to Greetdelay16 the ESMTP connection. Spamcontrol provides three solutions for the Though the idea points back to tarpitting, (fortunately currently not so virulent) poten- in spite of the million Bot net PCs com- tial infections of the target systems: manded to send spam mails, this mecha- 1. The ’Warloard’ algorithm allows to re- nism is surprisingly efficient and easy to ject particular attachments based on their use. MIME type and Loader recognition in • Currently, the alternative Greylisting is 15 wire-speed . not supported by Spamcontrol, though 2. A seamless integration of the AV scanners, some solutions exist for Qmail. Actually, like in particular ClamAV, can be facili- I am not a friend of Greylisting, it’s effect tated by means of the ’Qmail High Perfor- on uncontrollable delivery latency and in mance Scanner Interface’ QHPSI. Here, particular it’s substantial coding require- the AV scanner directly access the email in ments, and potential failures. the Qmail queue. Since most AV scanners • To recognizes spam mails on behalf of it’s have a built-in understanding of BASE64 reputation and/or content is the tradi- attachments and also support scanning of tional regime of SpamAssassin17. archives like ZIP, there is simply no par- The complex operations SpamAssassin ticular reason why sophisticated ’staging performs, requires (a) CPU power, (b) fast areas’ are required. disk access (c) network connectivity. Item 3. Spamcontrol also supports the (b) is currently a limiting factor. QUEUE EXTRA scheme, allowing Spamcontrol comes with wrapper-script the replacement of qmail-queue by a qmail-queue.scan which allows to per- wrapper in the first place, to do virus and form all virus and spam scanning activities spam scanning. commonly on a RAM disk, rather than on the real disk, speeding up those tasks sig- nificantly in addition with a substantial 3.5 Anti-Spam tools reduction of I/O load. Though Spamcontrol provides some advanced means to control the SMTP envelope, spam 3.6 QMQ sender have (mostly) accommodated to that and we are left with some basic means: One important lesson of today’s email process- ing is the use of multi-tier systems. Often, • Employing tcpserver or sslserver, the the last member in the chain is a Microsoft existence of a qualified DNS record of Exchange server, hosting the mailboxes of the the sender can be checked. This infor- users, while the front-end servers (close the In- mation can be uses to allow a badmail- ternet) are recruited from systems operating fromunknown and badmailfromwell- under the Unix OS. However, those Exchange known filtering. systems often need to be serviced individu- ally and protected in particular from spam and • rblsmtpd allows the lookup into several virus storms. Relay Black Lists RBL. 16http://www.fehcom.de/qmail/Greetdelay.pdf 15published in: c’t 11/2004 ’Exe und Hopp’ 17http://spamassassin.apache.org/
5 The ’Qmail Multiple Queue’ option (QMQ) of Spamcontrol provides an appropriate solu- tion for this task. Here, one Qmail system (us- ing Spamcontrol) is responsible for the Internet connectivity, while vanilla Qmail instances are used to transmit emails from and to the target email systems. All Qmail systems may com- municate with the QMTP protocol instead of SMTP. Since the 2nd line systems can be ad- justed to the required Service Level Agreements SLAs, this results practically in a predictable delivery delay to the upstream MTA. By the same token, a dedicated Bounce Figure 1: Setup of qmail-spamcontrol options Host can be setup, which not only releases the standard Qmail queue from delivering NDR’s (most to none-existing user account in case • ’sysutils/daemontools’ (0.76 15) provided of spam), but also helps to avoid accidentally by Peter Pentchev coming with the Gerrit blacklisting the standard Qmail instance. Pape’s man-pages.
• ’dns/djbdns’ (1.05 13) again from Peter 4 FreeBSD Qmail Ports Pentchev supporting in addition IPv6.
Since it’s Dan Bernstein’s concept18 instead of • ’security/checkpassword’ (0.90) as a po- having an ’all-inclusive’ solution rather to use tential PAM to access /etc/passwd. dedicated and mutually ’untrustworthy’ ser- • ’mail/qmailanalog’ (0.70 2) package to vices, a complete installation of Qmail depends statistically analyse the qmail-send log. on very many components which are now avail- able as ports for FreeBSD (mostly done by Re- • ’mail/qmailmrtg7’ (4.2 4) port from Alex nato): Dupre using Inter7’s19 presentation of To- bias Oetiker’s MRTG20. • ’mail/qmail-spamcontrol’, the most recent port of Spamcontrol (2.6.23 1) is available Apart from ’qmail-spamcontrol’ there exist thru portsnap. other major enhancements for Qmail: ’qmail- tls’ is based upon Frederik Vermeulen’s TLS • ’sysutils/ucspi-tcp’ (0.88 2), including implementation21 and ’qmail-ldap’ providing man-pages and optional IPv6 support Andre Oppermann’s LDAP22 solution. How- (among) other. ever, due to their complexity those ports con- flict with each other. • ’sysutils/ucsp-ssl’ (0.70 1) contributed by End of 2007, Dan Bernstein has released al- David Thiel and including my patch for most all of his software into the public domain STARTTLS support. 19http://www.inter7.com/ 18D.J. Bernstein: ’Some thoughts on 20http://tobi.oetiker.ch/hp/ security after ten years of qmail 1.0’ 21http://inoa.net/qmail-tls/ http://cr.yp.to/qmail/qmailsec-20071101.pdf 22http://www.nrg4u.com/
6 which not only allows (private) modifications one (E)SMTP session is used, even though but also permits the distribution of the mod- the email targets many recipients on the same ified sources without conflicting with any li- system; which has been discussed with many cense restriction. pros&cons in the past. However, this behav- Let’s focus on two important add-ons for ior guarantees that each email is stored indi- qmail-spamcontrol, Renato has included. vidually in the Message Store. This – in turn – requires that the process qmail-send cares 4.1 SPF enhancement about each and every email and updates it’s status in the queue concurrently: in it’s spare The Sender Policy Framework23 adds repu- tation to the sending MTA by means of either time. However, on a busy email system this the specific SPF Resource Record or a stan- conflicts with other tasks of qmail-send and dard TXT Record: may result in a race condition named ’silly qmail syndrome’. example.com. 3600 IN SPF "v=spf1 mx -all" Since the qmail-smtpd process is light- Both, the mail administrator and the DNS weight, a mailserver connected to the Internet administrator have to carefully define and can easily service 5000 concurrent SMTP con- setup those records. On the other hand, it is nections, and of course needs to get rid of those task of SMTP server to emails not keeping those in the Message Store 1. query this information, for too long25. There are a few solutions to pre- 2. perhaps to reject mails in case of inade- vent/overcome this situation: quate SPF settings, or 1. The more sending channels Qmail 3. to forward this information to external (qmail-remote) has, the faster the spam checkers, like SpamAssassin, and emails can be released from the MS. Dan 4. to report those findings in an additional Bernstein has initially foreseen a very Received-SPF: header. conservative limit for a remote concur- rency of 255, while within the FreeBSD Within the qmail-spamcontrol port, qmail- Qmail port this is raised to 509. smtpd is responsible for those operations. Re- nato took the native (DJB-like) code from 2. Andre Oppermann has enhanced Qmail to Christophe Saout24. The resulting spfbehav- simply split qmail-send’s responsibility ior can be adjusted by means of control files – for handling those queue operations by and even more fine-grained – by environment means of the EXTTODO PATCH avail- variables defined for certain IP addresses or do- able as option in the port. main names. 3. Another possibility is, to setup Qmail 4.2 Concurrencies and Big-Ext- as a multi-tier system: QMQ. A script Todo ’multiple-queues.ksh’ (customizable via the file ’conf-qmq’) allows to configure up Since Qmail was written primarily as RFC 821 to 100 Qmail instances, which is the pre- MTA, it incorporates the idea of ’single mes- ferred way for Spamcontrol. sage transaction’. This means, for every email 25In this sense Greylisting impacts the book keeping 23RFC 4408 of a mailserver significantly, while Greetdelay puts the 24http://www.saout.de/misc/spf/ burdon on the network stack
7 5 Qmail ’Up and Running’ ? created before – , and
After the required ports have been installed, • some specific settings of svcscan logdir, there is unfortunately no-such-thing like a run- svscan lognum, svcscan logmax which ning Qmail. While checking the FreeBSD may be left to the defaults in the begin- handbook, section 28.4 (’Changing Your Mail ning. Transfer Agent’) provides some ideas about it and in particular references Postfix, there is 5.2 multilog practically no information about Qmail. While the script ’enable-qmail’ provides A busy mail server provides a lot of logging in- some hints at the end of the installation formation. If you run an ISP-like service, it is process, and within the created directory important to keep those logs for a certain time ’/var/qmail/boot/’ some start-up scripts exist, period, typically a few days. Those logs are they only allow a very basic set-up of Qmail. also required to provided monitoring based on Actually, they require the completely outdated MRTG and perhaps alarming. use of inetd and rather should completely re- In addition to the standard qmail-send log, moved from the port! Spamcontrol provides extensible logging for In addition, the recommended settings qmail-smtpd. This information is vital to of qmailsmtpdenable="YES" becomes com- identify in particular spam storms and other pletely obsolete in spite of supervise. attacks. On the other side, the qmail-spamcontrol While for low-volume mailservers sys- FreeBSD port supports the recommended logd may be an appropriate choice, a ’mailwrapper’ functionality while adding busy mailserver (we are talking about 1-10 the appropriate Qmail modules to the file mio/mails per day on a single server) mul- /etc/mail/mailer.conf. tilog is a must. Here, the logs of the indi- vidual daemons are written to particular di- So what is missing ? rectories with automated log-file cycling and backups. Easily 1 GByte logging information 5.1 supervise per qmail-smtpd instance can be written a We consider to run Qmail (and it’s compo- day, and are in particular handled. In case, Er- 26 nents) supervised. Having already installed win’s Newanalyse script together with the the Daemontools port, we need to enable it. qmailanaloge port is used, automatic reports While Dan Bernstein’s recommends the set- and backups are provided. In addition, every ting of csh -cf ’/command/svscanboot &’ single email processed by Qmail can be easily into rc.local, the FreeBSD port transaction-safe tracked. Typically, we keep the logs in directories • provides the script svscan within named /usr/local/etc and allows a customiza- tion by means of FreeBSD’s standard • /var/log/qmail-send/ /etc/rc.conf variables: • /var/log/qmail-smtpd/ • svscan enable="YES", • /var/log/qmail-pop3d/ • svscan servicedir defaulting to /var/service/ – which needs to be 26http://www.fehcom.de/qmail/newanalyse.html
8 and setting the ownership to ’qmaill’, MAXCONCURRENCY="100" Qmail’s dedicated log user. export HELOCHECK="A" This scheme can be extended to other ser- export QHPSI="clamdscan" vices as well. We keep in mind, that the size exec sslserver -sven -Rp\ -l $HOSTNAME\ of the attached filesystem and it’s speed is an -c $MAXCONCURRENCY\ important factor here. -x /var/qmail/etc/tcp.smtpd.cdb\ -u $QMAILDUID -g $QMAILDGID\ 5.3 run scripts 192.168.192.1 smtp\ rblsmtpd -WCr sbl-xbl.spamhaus.org\ In the supervise context a daemon or ser- /var/qmail/bin/qmail-smtpd\ vice is identified as (directory) name, while the cmd5checkpw true 2>&1 start script always is named ’run’27. Thus, for any of the serviced directories we need a par- Ouch. It’s obviously getting difficult to run ticular run script operating under the control a MTA! of supervise. However, unlike FreeBSD’s tra- ditional start/stop scripts (in particular those 5.4 Configuration & Customization coming with the qmail-spamcontrol port), run scripts simply catch signals from supervise In the Qmail universe, in particular qmail- and/or the command svc to operate and hence smtpd needs do not require arguments like ’start’ or ’stop’. 28 Gerrit Pape has released a nice collection of 1. site-specific configuration, providing the run scripts for daemons, like apache, postfix basic setup of the daemon, resource limits, and many others. bindings and adjacent programs to call. The logging to multilog is facilitated as ’co- These settings stay constant as long as the process’ simply piping the information avail- process is running. Environment variables able at FD1 and FD2 to this process placed in can be mapped into a directory by means an adjacent ./log/ directory. of Daemontools’ envdir program, and Here are two samples of my own (slightly trimmed) run scripts: 2. service-specific customizations allowing qmail-send: run-time modifications, for instance SMTP envelope filters and in particu- #!/bin/sh lar the cdb which defines the rules for exec env - PATH="/var/qmail/bin:$PATH"\ tcpserver or sslserver. qmail-start ./Maildir/
qmail-smtpd: Though Qmail – in it’s basic setup – can very well operate with a minimum of configuration #!/bin/sh & customization, in practice the more settings QMAILDUID=‘id -u qmaild‘ QMAILDGID=‘id -g qmaild‘ are provided, the more are used (and of course HOSTNAME=‘hostname‘ are often only very barely documented). The FreeBSD port ’mail/qmailadmin’ seems 27Frankly, this is the opposite idea w.r.t. System V ’s run-level scripts to be found in one container to solve that problem; but too bad, this pack- 29 (/etc/init.d) containing alpha-numerically ordered age is only suited for Inter7’s Vpopmail . symlinks to the start/stop scripts 28http://smarden.org/runit/runscripts.html 29http://www.inter7.com/?page=vpopmail
9 5.5 Twisted Edges In practice – due to the dependency of Qmail with for instance djbdns and the queried LDAP servers for email address validation the virus scanning, SpamAssassin and other tools – a suitable Qmail system becomes even more complex. Thanks to supervise and multi- log, we guarantee high-availability and non- repudiation of email transactions, providing a soft pillow for administrators, though not solv- ing the case of
• configuration management and
• alarming (incident management).
6 Summary
Running the ported Qmail on FreeBSD pro- vides – under the condition of careful configu- ration & customization – a high-available and high-performance MTA. On current hardware, 1-10 mio/emails per day can be handled with full virus and spam scanning on a single ma- chine with low thruput latency. Due to the concept of virtual domains, Qmail is also very well suited for large-scale end-user support. However, the final bill includes substan- tial positions regarding the preparation of the Qmail system to fit seamlessly into the FreeBSD universe.
10