A case study on building a mail archive with postfix Cologne 26.09.2018 1 What is a Mail Archive ? • A central system that keeps a copy of all incoming and outgoing e-mails. • Internal or external mail server • Archive is rather internal; can be integrated in/part of the mail server Incoming/ Incoming/ Mail server outgoing Mail server outgoing mail mail Mail archive Mail archive 2 Features of a Mail Archive • Archive all mails for later retrieval • All mails: new, current, different accounts • Save storage, reduce workload: Release mails from the mail server • Search • Keep structure of folders, meta data • Simplify backup for externally managed mail • Protection against failover (service outage, data loss) • Protect against general legal risk • Meet the EU‘s General Data Protection Regulation (GDPR) 3 Mail Terminology Mail User Agent (MUA) SMTP Mail Delivery Agent (MDA) POP3/IMAP POP3/IMAP Thunderbird, Roundcube,… Cyrus, Dovecot Internet - Some destination mail server or Mail Transport Agent (MTA) - receiving mail from this server SMTP Postfix 4 Considered Tools / approaches Forwarding approach Synchronization approach Postfix BCC feature Dovecot‘s dsync / doveadm backup Imapsync v3.1.8 v2.2.27 v1.882 • Postfix is a free and open- • Dovecot is an open source IMAP • Imapsync is an IMAP transfers source mail transfer agent (MTA) and POP3 email server for tool. The purpose of imapsync is that routes and delivers Linux/UNIX-like systems to migrate IMAP accounts or to electronic mail. • Debian package dovecot-imapd backup IMAP accounts. • Debian package postfix available • No package but easy instructions here: https://imapsync.lamiral.info/INSTA LL.d/INSTALL.Debian.txt • Did not test cyrus backup (current version 3.0.8) • Feature is not compiled • Cyrus Version 3.0.8 Documentation: “Cyrus Backups are a replication-based backup service for Cyrus IMAP servers. This is currently an experimental feature.” 5 Maildir Format (vs. mbox) • Each mailbox folder is a directory, each message a file (mbox uses single file) • Index for each folder (search, detect duplicates) • Improved efficiency • Generally the preferred format 6 Mail archive user interface • Mail client application (Thunderbird, …) or Webmailer (Roundcube, …) 7 Rating Table Postfix doveadm doveadm doveadm imapsync bcc backup 2-way sync 1-way sync Installation ? ? ? Documentation Configuration Feasibility/ Integration Useful for archive Log File Performance Legal perspective ? ? ? ++ = very good + = good O = ok - = not so good -- = bad 8 Postfix BCC feature: a) internal mail server setup Internal Mail Server: Postfix, Dovecot Retrieve mail IMAP SMTP SMTP Incoming/outgoing Send mail mail • Fully self-hosted mail server • Can be internal or external 9 Postfix BCC feature: Definition of rules 1. Create accounts for mail archive: #adduser user1_rcv, user2_rcv, … 2. Create two files bcc_archive_rcv and bcc_archive_snd The two files define to which account incoming and outgoing mail is archived: # /etc/postfix/bcc_archive_rcv [email protected] [email protected] [email protected] [email protected] # /etc/postfix/bcc_archive_snd [email protected] [email protected] [email protected] [email protected] 10 Postfix BCC feature: Include the rules in postfix configuration 3. Insert two lines in /etc/postfix/main.cf: # file for receiving mail rules recipient _bcc_maps = hash:/etc/postfix/bcc_archive_rcv # file for sending mail archiving rules sender_bcc_maps = hash:/etc/postfix/bcc_archive_snd 4. Translate the hash tables: # postmap /etc/postfix/bcc_archive_snd # postmap /etc/postfix/bxx_archive_rcv 5. Restart postfix 11 Postfix BCC: b) external mail server Internal Mail Server: Postfix, Dovecot, fetchmailrc External Mail Server Receive mail IMAP Fetch mail POP3 SMTP SMTP Send mail Send outgoing mail to external gateway • Fetch mails from external mail server with fetchmailrc (mails are stored locally) • Configure Postfix to use the external mail server for sending mail • User sends/ retrieves mail to/ from internal server • Incoming and outgoing mail is archived with Postfix BCC feature 12 Postfix BCC: b) external mail server - features • own mail server but without public IP, external provider takes care of domain, spam, … • Security: no open port • Poll multiple external mail accounts: central collection point • Better performance due to local network but VPN might be needed Internal Mail Server: Postfix, Dovecot, fetchmailrc External Mail Server Receive mail IMAP Fetch mail POP3 SMTP SMTP Send mail Send outgoing mail to external gateway 13 Configuration of fetchmailrc # /etc/fetchmailrc poll mailserver1.tld protocol POP3 user username1 password password1 to localuser1 ssl poll mailserver2.tld … • Can poll multiple accounts 14 Configuration of external mail gateway/ relay in Postfix # a few more lines in /etc/postfix/main.cf relayhost = [mail.domain.com]:25 # for authentification: smtp_sasl_auth_enable = yes smtp_sasl_password_maps = hash:/etc/postfix/relay_passwd #/etc/postfix/relay_passwd smtp_sasl_security_options = noanonymous [mail.domain.com]:25 username:password 1) Edit main.cf 2) Create the file /etc/postfix/relay_passwd 3) Create the hash database file for postfix sudo postmap /etc/postfix/relay_passwd sudo service postfix restart 15 Postfix BCC: rating • Easy to implement if postfix is already used (internal mailserver) • Otherwise requires intrusive modification of the mail system • External mail server: In case of a fail out mails still do not get lost • Folders are NOT preserved !! 16 Rating: Postfix Postfix doveadm doveadm doveadm imapsync bcc backup 2-way sync 1-way sync Installation ++ Documentation ++ Configuration + Feasibility/ Integration ++/O Useful for archive + Log file O Performance ++ Legal perspective ++ = very good + = good O = ok - = not so good -- = bad 17 Synchronization Approach • Two mail servers one is source, the other is the archive • Testbed with local network • IMAP, pull from archive-side • Start with dovecot, then imapsync Mail server Archive mail server (source) (destination) IMAP pull 18 dovecot‘s synchronization utilities: info of the documentation • doveadm backup performs one-way synchronization. If there are any changes in the destination they will be deleted, so the destination will look exactly like the source. • doveadm sync performs two-way synchronization. It merges all changes without losing anything. Both the mailboxes will end up looking identical after the synchronizations finished. • doveadm sync -1 performs one-way synchronization, but it merges the changes in destination without deleting 19 doveadm -Dv \ -o imapc_host=192.168.8.230 \ doveadm backup -o imapc_user=st-alex \ -o imapc_password=xxxxxxxx \ -o imapc_ss=starttls \ backup -R -u rein imapc: • -R: call from archive and pull the mail from source • Source and destination exactly look the same • Changes in the destination are overwritten • Destination should be empty on first call 20 doveadm 2-way sync doveadm -Dv \ -o imapc_host=192.168.8.230 \ -o imapc_user=st-alex \ -o imapc_password=xxxxxxxx \ -o imapc_ss=starttls \ sync -R -u rein imapc: • Again source and destination look the same • But … 21 doveadm 2-way sync • … you can make changes on both sides • Delete testmail 1 on source • Delete testmail 2 on destination • Receive testmail 6 at destination 22 Result of doveadm 2-way sync • Work on both sides • No „master“ side (rsync would need one) • Index file to keep track of the changes • Also syncs metadata/flags (read flag, deleted flag, …) 23 Doveadm sync -1 (one way sync) doveadm -Dv \ -o imapc_host=192.168.8.230 \ -o imapc_user=st-alex \ -o imapc_password=xxxxxxxx \ -o imapc_ss=starttls \ sync -1 -R -u rein imapc: • Some new mail in the source • Some „old“ mail in the archive 24 Doveadm sync -1 (one way sync) • Source stays the same • New mail is merged to the archive 25 Doveadm sync -1 (one way sync) Before sync After doveadm sync -1 Receive new mail in the source New mail is synced to the archive, old mail in the archive is not deleted Delete mail in the source Mail is not deleted in the archive Delete mail in the archive Mail stays deleted in the archive Change a mail flag in the source Change is not synced to the archive Change a mail flag in the archive Change is not synced to the source Receive new mail from Postfix in the archive Doveadm gets confused, new mails from source are duplicated • Mail is synched/copied from the source to the archive • Synch does not delete or change any mail in the archive • Changes in the archive are not taken back 26 Doveadm: useful for a mail archive? • Doveadm backup, 2-way sync: Snapshot of mail folders • Snapshot has to be completed with a general backup strategy and tool (Bacula, Bareos, Amanda, …) • Doveadm 1-way sync: does not delete mail, may serve as a standalone mail archive • 2-way sync is more suitable if you (also) need a failover mail system or need to work on both sides • Problem of backup granularity: archive might not be complete 27 Doveadm feasability / requirements for 1-way and 2-way sync • Use dovecot on source and destination • Use correct version 2.2 on both sides 28 Rating: Synchronization approach Postfix doveadm doveadm doveadm imapsync bcc backup 2-way sync 1-way sync Installation ++ ++ Documentation ++ O Configuration + O Feasibility/ Integration ++/O + - Useful for archive + + O + Log file O - Performance Legal perspective ++ = very good + = good O = ok - = not so good -- = bad 29 imapsync • Sync two mail accounts via IMAP • Configuration/options are very easy: ./imapsync \ --host1 test1.lamiral.info