Design and Management of Service

Source : homepage.ntu.edu.tw/~jsc/2005-mail.ppt Outline

 Introduction to the architecture and operation of SMTP  Design of a suitable email system – Webmail solutions  and simple configuration samples  Spam and virus filtering  Conclusion

2 Overview

 Electronic mail service has already evolved into one of the major Internet applications.  It is not only fundamental, but also a must.  Users may become impatient when mails were delayed, not to mention failed to access their . – Imagine we meet the situation of power failure or cut of water supply

3 Architecture of a Simple Mail System

 Consists of the following components – MTA - Mail transfer agent  Sending and forwarding emails  Server end – MDA - Mail delivery agent  Delivering emails to recipients’ mailbox  Server end – Pop3/Imap4 Daemons  For users to download their mailboxs  Server end – MUA - Mail user agent  Reading and composing emails 4  Client end Architecture of a Simple Mail System Protocols Used for Mail System

 Protocols – For computer programs to communicate with each other – Similar to languages that human beings speak  SMTP – Simple Mail Transfer Protocol – Too simple to provide any “advanced features”  Authentication  Authorization  POP3 – version 3 – Simple  IMAP4 – Internet Message Access Protocol version 4 – Fully compatible with internet message standards, e.g. MIME. – Allow messages to be accessed from more than one computer. – Provide support for online, offline, and disconnected modes. 6 – Multiple and share folders. Mail Forwarding Between Servers How to Find the Way to the Destination?

 How do we find the way to [email protected]?

8 DNS: The Key to All Internet Services

 Query DNS server by the address part of .([email protected]) 1. Query for MX records 2. Query for A records 3. Give up!  Not required to exist a real machine hosting the address.(ntu.edu.tw)  Adding MX records to improve query performance and to reduce unnecessary query overhead.

9 DNS: Query Results

 Precedence value of MX records – Lower values with higher priority

10 DNS: Query Results

 When MX record is absent, second query is required to issue.

11 Features Required By Modern Mail Systems

 This is what end users care.  Ease of use interface – IMAP4 – Webmail  Security – Authentication – Encrypted transmission  Spam and virus filtering – The only effective way to prevent from virus infection is to remind and force your users to install antivirus softwares. 12 – Stop using M$ systems is alternative solution. Make the Right Decision For the Promise of Performance

 MTA – Postfix  Our suggestion! –  Not scalable!  Lack of new features.  Not efficiency in resource management.  Rich of documents and tutorials –  Termination of development. –  Immature. – M$ Exchange  Seeking trouble for yourself?  POP3/IMAP4 – UW-IMAP – Courier-IMAP  Supports Maildir format. –  Supports both and Maildir format.  Still under development. – Cyrus  Proprietary spool format. 13  Not easy to convert between other formats. Webmail: The Trend of Browser Based Mail Reader

 Your determination depends on the system architecture.  IMAP4 based – Access the mailbox via IMAP4 protocol locally or remotely. – Flexible, scalable – Speed may be limited by protocol overhead and network transmission. – Horde IMP project  Locally direct access – Access the mailbox locally, proprietary protocol. – Could only run on local mail storage server. – NFS could be applied when deploying multiple servers – Extremely fast!! iff system not overloaded. – Not scale well when you want to expand. – Openwebmail  Excellent project developed by NCKU, Taiwan  Marvelous support to Chinese encodings  Only support mbox format currently 14 Webmail Architecture – IMAP4 based Webmail Architecture – Locally Direct Access Postfix: The Modern and Advance MTA

 http://www.postfix.org  Developed by Wietse Venema  Features – Easy replacement of existing sendmail system by binaries with same file name and functionality . e.g. sendmail, newaliases – Human readable configuration file – Multiple small programs with limited execution privilege – Multiple queues to avoid block of normal mails – Backend database lookup table supported – Better support for virtual host – Virtual user system supported – Content filtering interface 17 – Simple spam check/block mechanism Postfix - the Big Picture

 http://www.postfix.org/big-picture.html – Yellow ellipsoids are mail programs. – Yellow boxes are mail queues or files. – Blue boxes are lookup tables.

18 Postfix Installation

Linux – apt-get install postfix-tls libsasl7 libsasl-modules-plain courier-imap  Redhat/Fedora Linux – rpm -ivh postfix-2.2.x.i386.rpm – rpm -ivh cyrus-sasl-2.1.21.i386.rpm  BSD – Use the ports system – /usr/ports/mail/postfix – /usr/ports/security/cyrus-sasl2 19 – /usr/ports/mail/courier-imap Post Installation

 Postfix Configuration – master.cf  Similar to inetd.conf  Control the behavior of small programs – In contrast against sendmail, with one binary and one config file – main.cf  The main configuration of the mail system – In general cases, no modification is required for a simple setup.  Commands to control the postfix program – postfix start – postfix stop 20 – postfix reload Test For Receiving Email

 Send an email to yourself.  Check system logs – /var/log/maillog  An simple test from [email protected] to [email protected]

21 Frequently Used Configuration Options In main.cf

 myhostname = mail.cc.ntu.edu.tw  home_mailbox = Maildir/  mydomain = cc.ntu.edu.tw  message_size_limit = 20971520  myorigin = $mydomain  smtpd_recipient_limit = 1000  mydestination = $myhostname, localhost.$mydomain – On a mail domain gateway, you should also include $mydomain.  mynetworks_style = host  relay_domains = $mydestination

22 SMTP/Authentication

 Traditional mail servers could only determine their relay policy based on connected ip addresses.  Insufficient in face of the modern network – DHCP, NAT – ADSL and dialup users – Roaming users  Mobile, wireless  SMTP Authentication – Username/password authentication for each mail delivery. – Reduce the chance to relay spam mails for anonymous. 23 Email Headers

 Envelope header – Negotiated between mail servers – Typical tricks that SPAM mail plays  Why do I receive mails that are not destined to me?  Content Text – Mail servers will not examine the content in general cases.

24 Real World Snail Mail Example

 Postman could only deliver the mail by envelope information.  Sender address is not verified when email delivery. – Trick of SPAM mails.  Content may be irrelevant or conflict with envelope.

25 Virus and Spam Mail

 Virus mail – Contain virus infected attachment – How do we determine if the mail is virus infected?  Simple and easy. Base on the inspection and judgment of antivirus software.  Spam mail – UCE/UBE  Unsolicited Commercial Email  Unsolicited Bulk Email – How do we determine if the mail is a Spam mail?  Difficult. It’s not a yes/no problem. 26  One person’s meat may be another’s poison. Issues When Dealing With SPAM

 Do not block or discard users’ mail unless you have the authorization and delegation.  The only safe and acceptable assistant you can provide is to score and tag each mail and let your users to do the rest themselves.  To warn or not to warn the senders/recipients is a dilemma! – Virus and spam senders nowadays always fake the from address. – System notification mails may be another kind of SPAM. The result will be blamed by your users and blocked by 27 other mail servers. Greylisting

 Recently proposed (2003) method to block spam and virus mails.  RFC requires MTA to be capable to support queue and retry of temporarily undeliverable mails. – Spam and virus always fire and forget. – For efficiency concern, usually they do not retry.  How it works? – In brief, block every delivery at its first try. – For each mail delivery, check if the the triplet(source ip, sender, recipient) was seen recently.  If yes, pass the mail.  If not, reject the delivery with “450 Please retry later.” 28 Greylisting

 Drawback – Mail delivery will suffer a delay ranging from several minutes to hours.  Depending on the remote MTA configuration.  Risk – Sites that do not retry their mail delivery.  Some bank notifications were observed to try only once.  Mail sending programs that process the SMTP transaction themselves without calling the “sendmail” MTA interface.  Suggestion – Whitelisted known hosts to prevent from delay and loss of mail. 29 Greylisting

 Benefit – Effectively reduce the volume of spam and virus. – Effectively reduce both cpu and disk load of servers.

30 Softwares To Block/Filter Spam and Virus

 Postfix – Content filter interface  Postgrey – Greylisting policy server  Amavisd-new – http://www.ijs.si/software/amavisd – http://www.ijs.si/software/amavisd/README.postfix  Clam Antivirus – http://www.clamav.net – A GPL virus scanner  SpamAssassin – http://spamassassin.apache.org 31 Software Architecture for Filtering

32 Amavisd-new

 A high performance interface between MTA and content checkers. – Calling external antivirus programs to do virus scanning. – Calling external spamassassin program to do spam level determination. – CPU intensive workloads. – Can be flexibly configured to pass, discard, or quarantine mails based on user defined policy.  Pass spam mails with score > 10 with subject prepended the *** SPAM *** keyword.  Quarantine spam mails with score > 20.  Discard spam mails with score > 30.  Quarantine virus mails. 33 Spamassassin

 Spam level scoring software.  Rich set of tests to identify various spam signatures. – Keywords, bad headers, encodings  Use bayesian analysis to help scoring. – Training the bayesian database using know spam and ham mails. – Default to enable the auto-learn feature.  Calling external programs to check if the mail was a known spam. – Use hash of mail content as the query key. 34 – Razor, DCC, Pyzor. Spamassassin

 RBL(realtime black list) look up based on sender ip address. – RBL may contains too many ill-administrated sites. – Use the result as an addition of spam score. – Do not block remote sites depend solely on RBL.  SURBL(Spam URI realtime black list) look up based on the URIs within the content of mail. – Spammers may keep changing their sending IP addresses. – The URIs in the content may be the final 35 destination the advertisement want people to visit. Make Use of the Filter Rules on your MUA

 Outlook Express – Can only filter based on limited headers.  Subject, Sender, etc.  Configure to move mails having subjects beginning with *** SPAM *** to an SPAM folder.  Outlook, Netscape, Thunderbird – You can filter mails based on any headers. – Determine your own spam level threshold referring the X-Spam-Level: ****** 36 Conclusion

 Make use of public domain, open source software to construct your own mail system.  Official web sites, mailing lists, and google are always your best friends.  Feedback and contribute as possible as you can.

37 Reference :

 homepage.ntu.edu.tw/~jsc/2005-mail.ppt

38