Mail Protocol, Postfix and Mail security HowHow Email Email Appears Appears to to Work Works
2 HowHow Email Email ReallyReally Works Works
3 Message Format
• Envelop – Routing information for the "postman" • Message Header – Sender – Recipients (simple, lists, copies, blind copies) – Other fields of control (date, subject) • Message Body – Free text – Structured document (i.e.: MIME) Message Format
From: GZ Kabir
Hi Kong, I'm going to be running a course on ...
ü Format was originally defined by RFC 822 in 1982 ü Now superseded by RFC 2822 ü Message consists of ü Header lines ü A blank line ü Body lines Message Format • Embedded MUA uses inter-process call to send to MTA • Freestanding MUA uses SMTP to send mail
• Headers added by the MUA before sending From: GZ Kabir
To: D iep K ong < [email protected] > Cc: Moin < [email protected] > Subject: How Internet mail works
Hi Kong, I'm going to be running a course on ... Mail Delivery Agent (MDA) / Mail Transfer Agent (MTA)
• Headers added by MTAs
From: GZ Kabir
To find the recipient's IP address and mailbox, the MTA must drill down through the DNS system, which consists of a set of servers distributed across the Internet beginning with the root name servers
root servers refer requests for a given domain to the root nameservers that handle requests for that tld – MTA can bypass this step because it has already knows which domain nameservers handle requests for these .tlds e.g. bofh.im asks the appropriate DNS server which Mail Exchange (MX) servers have knowledge of the subdomain or local host in the email address
DNS server responds with an MX record: a prioritized list of MX servers for this domain
To the DNS server, the server that accepts messages is an MX server. When is transferring messages, it is called an MTA.
MTA contacts the MX servers on the MX record in order of priority until it finds the designated host for that address domain
sending MTA asks if the host accepts messages for the recipient's username at that domain (i.e., [email protected]) and transfers the message Firewalls, Spam and Virus Filters
Firewalls, An email encountering Spam a firewall and Virus Filters may be tested by spam and • An email encounteringvirus filters beforea firewall it is allowed may be tested byto spam pass inside and thevirus firewall filters before it is allowedfilters test to to pass see if inside the the firewall message qualifies as spam or malware • filters test to see if the message qualifies as Ifspam the message or malware contains malware, the file is usually • If the messagequarantined contains and malware, the sender is the file is usuallynotified quarantined and
the sender isIf the notified message is identified as spam, it will probably be • If the messagedeleted is identified without notifying as spam, the it will probablysender be. deleted without notifying the sender. 1 6 Troubleshooting Email Issues
• transient failures – If a transient error occurs, the MTA will hang onto the message, periodically retrying the delivery until it either succeeds or fails, or until the MTA decides that the transient issue is really a permanent condition. • permanent failures – If the MTA cannot deliver the message (it has received a fatal error message or failed to complete the transfer after repeated attempts), it bounces the message back to the sender. If the sender is a mailing list, the bounce may be handled by automated bounce-handling software.
Concept of Mail Protocols Component of Email system ❏ Mail Transport Agent/Message Transfer Agent (MTA) ❏ Mail User Agent (MUA)
❏ Mail Delivery Agent(MDA) MTA
The actual mail transfer is done through message transfer agents (MTAs). To send mail, a system must have the client MTA, and to receive mail, a system must have a server MTA. The formal protocol that defines the MTA client and server in the Internet is called Simple Mail Transfer Protocol (SMTP) SMTP uses commands and responses to transfer mail between an MTA client and MTA server. SMTP Mail transfer Flow MTA connection setup MTA Connection Setup..Contd.. Sender opens TCP connection with receiver □ Once connected, receiver identifies itself □ 220 service ready □ Sender identifies itself □ HELO
Each message transfer has the following phases: • One MAIL command, identifies originator •Gives reverse path to use for error reporting •Receiver returns 250 OK or appropriate fail/error message • One or more RCPT commands, identifies recipients for the message • Each recipient identified by a separate RCPT • Separate reply for each recipient (250 OK etc.) • One DATA command transfers message text • End of message indicated by line containing just period (.) MTA connection termination MTA connection termination Sender sends QUIT and waits for reply
□ Then initiate TCP close operation
□ Receiver initiates TCP close after sending reply to QUIT Optimization If message is sent to multiple users on a given host,it is sent only once.
❖ Delivery to users handled by destination host
If Multiple messages are ready for given host,a single TCP connection can be used.
❖ Saves overhead of setting up and termination of connection Possible Errors ❖ Host unreachable ❖ Host out of operation ❖ TCP connection fail during transfer ❖ Faulty destination address
User error Target user address has changed Redirect if possible Inform user if not Sender re-queue the mail - will retry till a configurable period of time SMTP protocol reliability ❖ TCP connection is used to transfer mail from sender to receiver over TCP connection ❖ Attempts to provide reliable service ❖ No guarantee to recover lost messages ❖ No end-to-end ack to sender ❖ Error indication report not guaranteed SMTP receiver ❖ Accepts arriving message ❖ Places in user mailbox or copies to outbound message queue for forwarding ❖ Receiver must ● verify local mail destination ● deal with errors ❏ Transmission ❏ Lack of disk space SMTP status codes(DSN- Delivery status Notification)
Leading digits Indicates catagories 2XX-Positive Completion Reply(Successful)
3XX-Positive Intermediate Reply(Redirection)
4XX-Transient negative completion reply(Client error)
5XX-Permanent negative completion reply(Server error) SMTP status codes..Cont. Problems with SMTP
No inherent security
n Encryption Only uses NVT (Network Virtual Terminal) 7- bit ASCII format E-mails can be forged….. HELO mail.rose.edu MAIL FROM: [email protected] RCPT TO: [email protected] DATA From: Dr. Art Zenner To: Professor Richards Subject: CIT 2243
Professor Richards, By department decree all students in your “Introduction to Unix” class are hereby to be granted automatic A’s. Thank you, Dr. Art Zenner . QUIT Extensions to SMTP MIME – Multipurpose Internet Mail Extensions
n Transforms non-ASCII data to NVT (Network Virtual Terminal) ASCII data w Text w Application w Image w Audio w Video MIME and Base64 Encoding
If the internet is the information highway, then the path for email is a narrow tunnel n Only very small vehicles can pass trough Then how do you send a big truck through a small ravine? n You have to break it down to smaller pieces and transport the pieces through the ravine, and reassemble the truck MIME and Base64 Encoding
The same happens when you send a file attachment via email. This is known as encoding n the binary data (256 bits) is transformed to ASCII text (128 bits n allowing it to fit through the tunnel On the recipient's end, the data is decoded and the original file is rebuilt. Mail Transfer Agents (MTA)
MTAs do the actual mail transfers MTAs are not meant to be directly accessed by users. Other MTA’s are: n Postfix n Qmail n MS Exchange n CC:Mail n Lotus Notes n ….etc. Problems with simple SMTP
The first one relates to message length. Some older implementations cannot handle messages exceeding 64KB. Another problem relates to timeouts. If the Client and server have different timeouts, one of them may give up while the other is still busy, unexpectedly terminating the connection. Infinite mail storms can be triggered. For example, If host 1 holds mailing list A and host 2 holds mailing list B and each list contains an entry for the other one, then a message sent to either list could generate a never ending amount of email traffic unless somebody checks for it. ESMTP (RFC 2821)
To get around the problems with simple SMTP, extended SMTP has been defined in RFC 2821. Clients wanting to use it should send an EHLO message instead of HELO initially. If this is rejected, then the server is a regular SMTP server, and the client should proceed in the usual way. If the EHLO is accepted, then new commands and parameters are allowed. POP3 Mail access protocol:
POST OFFICE PROTOCOL [RFC 1939]
Simple and limited functionality,Consists of client software and Server software, Server performs user authorization
IMAP: Internet mail access protocol [Version 4] POP3 Retrieves messages from a mail server
Typically, messages are downloaded to your mail client, and deleted from the server
Designed for use with dial-up connections when people were intermittently connected
Listens on Port 110 (with Secure POP generally on port 995) POP3 Connection Establishment POP3 protocol session root@amberit ~]# telnet mail.amberit.com.bd 110 Trying 206.71.88.102... Connected to mail.amberit.com.bd. Escape character is '^]'. +OK Dovecot ready. user [email protected] +OK pass letmein +OK Logged in. list +OK 1 messages: 1 482 . retr 1 +OK 482 octets
□ Replies □ +OK □ –ERR IMAP Protocol session [root@amberit ~]# telnet mail.amberit.com.bd 143 Trying 206.71.88.102...
Connected to mail.amberit.com.bd Escape character is '^]'. * OK Dovecot ready. A1 LOGIN [email protected] A1 OK Logged in. A2 SELECTInbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * IMAP Protocol session..cont. OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. * 1 EXISTS * 0 RECENT * OK [UIDVALIDITY 1225333589] UIDs valid * OK [UIDNEXT 2] Predicted next UID A2 OK [READ-WRITE] Select completed. A3 FETCH 1 BODY[HEADER] * 1 FETCH (BODY[HEADER] {454}
Webmail (or web-based email) is any email client implemented as a web application running on a web server
• Online in nature • IMAP connections mostly • Secured HTTP (HTTPS) must • Can be hosted, e.g.: RoundCube, SquirrelMail, Horde, Zimbra, Rainloop, Kite, MailPile,iRedMail, etc. • Webmail Service providers : Gmail, Outlook, AOL, Yahoo, etc. • Privacy and Security Concern • Easy to configure, easy to host. Common Threat Landscape
• Eavesdropping • Spamming and Phishing • Spoofing • Malicious Email Attachments • Replying and Forwarding Issues • CC & BCC Issues
Postfix Short History
• Originally developed in the late 90s at IBM by Wietse Venema, author of security software (SATAN, TCPwrappers, ...), as ”IBM Secure Mailer” • Place under an Open Source license, and renamed ”Postfix” • Intended as a replacement for then insecure mail systems, such as Sendmail Design Goals
• Safety • Robustness • Performance • Modularity • Compatibility Safety
• Postfix makes it very hard to lose mails – many checks to ensure that mail has been written to disk or delivered • Back off mechanisms in case of repeated failure Security • Collection of daemons working together Doesn't use environment for communication • Very paranoid about input checking, all allocation is dynamic (avoiding buffer overflows) • chroot support out of the box for almost all processes & daemons • No data is ever exchanged directly between processes – all is done via IPC, and files on disk • Conservative resource usage Performance
• Designed to be fast from the ground up • Also behaves well with neighbors, doesn't flood them with mail, and instead uses a throughput adaptation • Will not block delivery for a message if one recipient domain fails Modular
• One program, one function All programs controlled from ”master.cf” • Many small programs working together, with limited privileges • Compatible with Sendmail's /etc/aliases and .forward conventions Features
• Virtual domains – domains and users are completely independent of system (UNIX) users • Aliases – sendmail compatible Rewriting – senders, recipients, globally RBL support (Realtime Blackhole Lists) support Content filtering using pipes, SMTP or milter • Support for arbitrary mail manipulation with policy services (custom programs talking to postfix) More Features
• Restriction classes Conditional filtering • Sender or recipient address verification (test email addresses before accepting mail from them) • TLS support Core Concepts : maps
• In postfix, everything is looked up in a map (table) • Maps can be in many formats or use many data sources: – hash/btree – regexp/PCRE – CIDR – NIS – LDAP, *SQL (user defined queries) Architecture Architecture Basic Postfix Configuration
• Two primary configuration files – main.cf • Main configuration file where all the subsystems are configured (smtp, smtpd, cleanup, routing, ...) – master.cf • File controlling how the ”master” process of postfix will launch all the necessary postfix daemons to perform mail routing, on-demand Other Configuration Details
• Reside in ”maps” mentioned earlier • Tables containing values and conditions, referred to from main.cf, controlling all aspects such as: – Virtual and local domains – Routing rules – Access control – Rewriting
– ... Configuration: postconf command
• postconf – used to view and edit configuration parameters • For changing the configuration, it is usually done vi editing ”main.cf” directly Some Basic main.cf smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination myhostname = group1.group01.net alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases myorigin = /etc/mailname mydestination = group1.group01.net, localhost.group01.net, , localhost relayhost = group1.group01.net mynetworks = 127.0.0.0/8 61.45.254.0/24 [2001:df0:a:4::]/64 [::ffff:127.0.0.0]/104 [::1]/128 mailbox_size_limit = 0 recipient_delimiter = + inet_interfaces = all inet_protocols = all Some Basic main.cf # TLS parameters smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key smtpd_use_tls=yes smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache Some Basic main.cf ## in the file /etc/mailname group1.group01.net
## in the file /etc/aliases postmaster: root sysadm: apnic