Design and Management of Email Service
Source : homepage.ntu.edu.tw/~jsc/2005-mail.ppt Outline
Introduction to the architecture and operation of SMTP Design of a suitable email system – Webmail solutions Postfix and simple configuration samples Spam and virus filtering Conclusion
2 Overview
Electronic mail service has already evolved into one of the major Internet applications. It is not only fundamental, but also a must. Users may become impatient when mails were delayed, not to mention failed to access their emails. – Imagine we meet the situation of power failure or cut of water supply
3 Architecture of a Simple Mail System
Consists of the following components – MTA - Mail transfer agent Sending and forwarding emails Server end – MDA - Mail delivery agent Delivering emails to recipients’ mailbox Server end – Pop3/Imap4 Daemons For users to download their mailboxs Server end – MUA - Mail user agent Reading and composing emails 4 Client end Architecture of a Simple Mail System Protocols Used for Mail System
Protocols – For computer programs to communicate with each other – Similar to languages that human beings speak SMTP – Simple Mail Transfer Protocol – Too simple to provide any “advanced features” Authentication Authorization POP3 – Post Office Protocol version 3 – Simple IMAP4 – Internet Message Access Protocol version 4 – Fully compatible with internet message standards, e.g. MIME. – Allow messages to be accessed from more than one computer. – Provide support for online, offline, and disconnected modes. 6 – Multiple and share folders. Mail Forwarding Between Servers How to Find the Way to the Destination?
How do we find the way to [email protected]?
8 DNS: The Key to All Internet Services
Query DNS server by the address part of email address.([email protected]) 1. Query for MX records 2. Query for A records 3. Give up! Not required to exist a real machine hosting the address.(ntu.edu.tw) Adding MX records to improve query performance and to reduce unnecessary query overhead.
9 DNS: Query Results
Precedence value of MX records – Lower values with higher priority
10 DNS: Query Results
When MX record is absent, second query is required to issue.
11 Features Required By Modern Mail Systems
This is what end users care. Ease of use interface – IMAP4 – Webmail Security – Authentication – Encrypted transmission Spam and virus filtering – The only effective way to prevent from virus infection is to remind and force your users to install antivirus softwares. 12 – Stop using M$ systems is alternative solution. Make the Right Decision For the Promise of Performance
MTA – Postfix Our suggestion! – Sendmail Not scalable! Lack of new features. Not efficiency in resource management. Rich of documents and tutorials – Qmail Termination of development. – Exim Immature. – M$ Exchange Seeking trouble for yourself? POP3/IMAP4 – UW-IMAP – Courier-IMAP Supports Maildir format. – Dovecot Supports both mbox and Maildir format. Still under development. – Cyrus Proprietary spool format. 13 Not easy to convert between other formats. Webmail: The Trend of Browser Based Mail Reader
Your determination depends on the system architecture. IMAP4 based – Access the mailbox via IMAP4 protocol locally or remotely. – Flexible, scalable – Speed may be limited by protocol overhead and network transmission. – Horde IMP project Locally direct access – Access the mailbox locally, proprietary protocol. – Could only run on local mail storage server. – NFS could be applied when deploying multiple servers – Extremely fast!! iff system not overloaded. – Not scale well when you want to expand. – Openwebmail Excellent project developed by NCKU, Taiwan Marvelous support to Chinese encodings Only support mbox format currently 14 Webmail Architecture – IMAP4 based Webmail Architecture – Locally Direct Access Postfix: The Modern and Advance MTA
http://www.postfix.org Developed by Wietse Venema Features – Easy replacement of existing sendmail system by binaries with same file name and functionality . e.g. sendmail, newaliases – Human readable configuration file – Multiple small programs with limited execution privilege – Multiple queues to avoid block of normal mails – Backend database lookup table supported – Better support for virtual host – Virtual user system supported – Content filtering interface 17 – Simple spam check/block mechanism Postfix - the Big Picture
http://www.postfix.org/big-picture.html – Yellow ellipsoids are mail programs. – Yellow boxes are mail queues or files. – Blue boxes are lookup tables.
18 Postfix Installation
Debian Linux – apt-get install postfix-tls libsasl7 libsasl-modules-plain courier-imap Redhat/Fedora Linux – rpm -ivh postfix-2.2.x.i386.rpm – rpm -ivh cyrus-sasl-2.1.21.i386.rpm BSD – Use the ports system – /usr/ports/mail/postfix – /usr/ports/security/cyrus-sasl2 19 – /usr/ports/mail/courier-imap Post Installation
Postfix Configuration – master.cf Similar to inetd.conf Control the behavior of small programs – In contrast against sendmail, with one binary and one config file – main.cf The main configuration of the mail system – In general cases, no modification is required for a simple setup. Commands to control the postfix program – postfix start – postfix stop 20 – postfix reload Test For Receiving Email
Send an email to yourself. Check system logs – /var/log/maillog An simple test from [email protected] to [email protected]
21 Frequently Used Configuration Options In main.cf
myhostname = mail.cc.ntu.edu.tw home_mailbox = Maildir/ mydomain = cc.ntu.edu.tw message_size_limit = 20971520 myorigin = $mydomain smtpd_recipient_limit = 1000 mydestination = $myhostname, localhost.$mydomain – On a mail domain gateway, you should also include $mydomain. mynetworks_style = host relay_domains = $mydestination
22 SMTP/Authentication
Traditional mail servers could only determine their relay policy based on connected ip addresses. Insufficient in face of the modern network – DHCP, NAT – ADSL and dialup users – Roaming users Mobile, wireless SMTP Authentication – Username/password authentication for each mail delivery. – Reduce the chance to relay spam mails for anonymous. 23 Email Headers
Envelope header – Negotiated between mail servers – Typical tricks that SPAM mail plays Why do I receive mails that are not destined to me? Content Text – Mail servers will not examine the content in general cases.
24 Real World Snail Mail Example
Postman could only deliver the mail by envelope information. Sender address is not verified when email delivery. – Trick of SPAM mails. Content may be irrelevant or conflict with envelope.
25 Virus and Spam Mail
Virus mail – Contain virus infected attachment – How do we determine if the mail is virus infected? Simple and easy. Base on the inspection and judgment of antivirus software. Spam mail – UCE/UBE Unsolicited Commercial Email Unsolicited Bulk Email – How do we determine if the mail is a Spam mail? Difficult. It’s not a yes/no problem. 26 One person’s meat may be another’s poison. Issues When Dealing With SPAM
Do not block or discard users’ mail unless you have the authorization and delegation. The only safe and acceptable assistant you can provide is to score and tag each mail and let your users to do the rest themselves. To warn or not to warn the senders/recipients is a dilemma! – Virus and spam senders nowadays always fake the from address. – System notification mails may be another kind of SPAM. The result will be blamed by your users and blocked by 27 other mail servers. Greylisting
Recently proposed (2003) method to block spam and virus mails. RFC requires MTA to be capable to support queue and retry of temporarily undeliverable mails. – Spam and virus always fire and forget. – For efficiency concern, usually they do not retry. How it works? – In brief, block every delivery at its first try. – For each mail delivery, check if the the triplet(source ip, sender, recipient) was seen recently. If yes, pass the mail. If not, reject the delivery with “450 Please retry later.” 28 Greylisting
Drawback – Mail delivery will suffer a delay ranging from several minutes to hours. Depending on the remote MTA configuration. Risk – Sites that do not retry their mail delivery. Some bank notifications were observed to try only once. Mail sending programs that process the SMTP transaction themselves without calling the “sendmail” MTA interface. Suggestion – Whitelisted known hosts to prevent from delay and loss of mail. 29 Greylisting
Benefit – Effectively reduce the volume of spam and virus. – Effectively reduce both cpu and disk load of servers.
30 Softwares To Block/Filter Spam and Virus
Postfix – Content filter interface Postgrey – Greylisting policy server Amavisd-new – http://www.ijs.si/software/amavisd – http://www.ijs.si/software/amavisd/README.postfix Clam Antivirus – http://www.clamav.net – A GPL virus scanner SpamAssassin – http://spamassassin.apache.org 31 Software Architecture for Filtering
32 Amavisd-new
A high performance interface between MTA and content checkers. – Calling external antivirus programs to do virus scanning. – Calling external spamassassin program to do spam level determination. – CPU intensive workloads. – Can be flexibly configured to pass, discard, or quarantine mails based on user defined policy. Pass spam mails with score > 10 with subject prepended the *** SPAM *** keyword. Quarantine spam mails with score > 20. Discard spam mails with score > 30. Quarantine virus mails. 33 Spamassassin
Spam level scoring software. Rich set of tests to identify various spam signatures. – Keywords, bad headers, encodings Use bayesian analysis to help scoring. – Training the bayesian database using know spam and ham mails. – Default to enable the auto-learn feature. Calling external programs to check if the mail was a known spam. – Use hash of mail content as the query key. 34 – Razor, DCC, Pyzor. Spamassassin
RBL(realtime black list) look up based on sender ip address. – RBL may contains too many ill-administrated sites. – Use the result as an addition of spam score. – Do not block remote sites depend solely on RBL. SURBL(Spam URI realtime black list) look up based on the URIs within the content of mail. – Spammers may keep changing their sending IP addresses. – The URIs in the content may be the final 35 destination the advertisement want people to visit. Make Use of the Filter Rules on your MUA
Outlook Express – Can only filter based on limited headers. Subject, Sender, etc. Configure to move mails having subjects beginning with *** SPAM *** to an SPAM folder. Outlook, Netscape, Thunderbird – You can filter mails based on any headers. – Determine your own spam level threshold referring the X-Spam-Level: ****** 36 Conclusion
Make use of public domain, open source software to construct your own mail system. Official web sites, mailing lists, and google are always your best friends. Feedback and contribute as possible as you can.
37 Reference :
homepage.ntu.edu.tw/~jsc/2005-mail.ppt
38