Renewal of the Email Services
IMAP,IMAP, SMTPSMTP && Co.Co.
WolfgangWolfgang FriebelFriebel Technical Seminar Zeuthen, 6.10.2009 Outline of the talk
SchematicSchematic viewview ofof thethe mailmail flowflow inin ZeuthenZeuthen WhatWhat willwill whenwhen bebe changedchanged ReceivingReceiving emailemail andand configurationconfiguration ofof mailmail clientsclients SendingSending emailemail SpamSpam andand mailmail filteringfiltering FrequentlyFrequently askedasked questionsquestions NextNext stepssteps
6. Okt. techn. Seminar 2 The mail flow in Zeuthen
SMTPSMTP forfor sendingsending mailmail external user SMTPSMTP AuthAuth ttoo sendsend fromfrom outsideoutside
SMTP Auth SMTP IMAPIMAP forfor readingreading mailmail IMAP externalexternal mailmail serverserver currentlycurrently alsoalso ext. mail server mailmail storestore int. mail server externalexternal serverserver performsperforms spamspam spam & SMTP tagging and virus scanning on a virus filter SMTP tagging and virus scanning on a mail store separateseparate machinemachine
IMAP UsersUsers cancan send/receivesend/receive emailsemails usingusing HamburgHamburg serversservers asas wellwell User 1 ... User n
6. Okt. techn. Seminar 3 Why a new IMAP Server
dovecotdovecot cancan handlehandle 10001000 activeactive usersusers onon aa singlesingle computercomputer UW-IMAPUW-IMAP (with(with mboxmbox format)format) allowsallows nono concurrentconcurrent accessaccess ConcurrentConcurrent accessaccess toto mailsmails cancan causecause lockinglocking oror lossloss ofof emailsemails MboxMbox formatformat limitlimit 2GB2GB reachedreached soonsoon Mailbox sizes DESY Hamburg distribution of mailboxes by size
SlowSlow accessaccess withwith mboxmbox formatformat 17 52 108 621 259 2088 157 270 Flexibility of dovecot 316 Flexibility of dovecot 379
ACL's for folders 3313 ACL's for folders Exchange compressedcompressed foldersfolders 1731 quotaquota UWimap
SituationSituation withwith developersdevelopers under 10 10M – 100M 100M – 250M 250M – 500M 500M – 1G 1G – 2G 2G – 5G over 5G
6. Okt. techn. Seminar 4 New IMAP Server
UW-IMAPUW-IMAP getsgets replacedreplaced byby dovecotdovecot ServerServer supportssupports additionaladditional foldersfolders besidesbesides INBOXINBOX (quota(quota ofof 11 GBGB inin additionaddition toto AFSAFS homehome directorydirectory quota)quota) mailmail quotaquota cancan getget displayeddisplayed usingusing check_inboxcheck_inbox (Linux)(Linux) nono moremore lockinglocking problemsproblems (AFS!!!)(AFS!!!) MuchMuch higherhigher speedspeed ofof emailemail accessaccess centralcentral mailmail filteringfiltering (sieve(sieve scriptsscripts onon imapimap server)server) centralcentral mailmail filterfilter byby defaultdefault movesmoves spamspam mailsmails intointo thethe junkjunk folderfolder UseUse ofof ownown filtersfilters possiblepossible butbut nono loginlogin onon imap,imap, hencehence uploadupload usingusing mailmail clientclient
6. Okt. techn. Seminar 5 Configuration changes for users
separateseparate serversservers forfor sendingsending ((mailmail)) andand readingreading ((imapimap)) ofof mailmail INBOXINBOX ofof allall usersusers willwill bebe onon imapimap insteadinstead onon mailmail moremore thanthan 5050 usersusers alreadyalready moved,moved, othersothers willwill followfollow untiluntil endend ofof OctOct newnew internalinternal serverserver forfor sendingsending ((mail1mail1)) isis operationaloperational nownow onon aa virtualvirtual machinemachine nono moremore accessaccess toto foldersfolders inin AFSAFS spacespace usingusing thethe IMAPIMAP serverserver foldersfolders inin AFSAFS accessibleaccessible asas locallocal foldersfolders onlyonly foldersfolders onon IMAPIMAP serverserver areare notnot inin AFSAFS spacespace
6. Okt. techn. Seminar 6 Comparison of UW-IMAP and dovecot
UW-IMAP dovecot
Server INBOX Folder (dirs) INBOX a1 b
1 file with many mails Mails (files)
AFS ~/mail ~/mail
Folder (files) a1 a2 b c Folder (files) a2 c
6. Okt. techn. Seminar 7 Move from mail to imap (1)
beforebefore OctoberOctober 15:15: movemove isis voluntary,voluntary, useruser isis sendingsending anan emailemail toto [email protected]@ifh.de whenwhen movemove shouldshould taketake place.place. OnOn requestrequest allall foldersfolders inin ~/mail~/mail inin AFSAFS spacespace cancan bebe copiedcopied toto thethe IMAPIMAP serverserver (max.(max. 500500 MB)MB) folderfolder namesnames containingcontaining certaincertain charschars (space,(space, .. )) cancan causecause problemsproblems useruser getsgets confirmationconfirmation mailmail sayingsaying thatthat thethe INBOXINBOX andand optionallyoptionally foldersfolders havehave beenbeen copiedcopied andand emailemail isis receivedreceived onon mailmail andand imapimap useruser cancan sendsend emailemail toto stopstop mailmail receptionreception onon mailmail AfterAfter atat mostmost 4848 hourshours mailmail receptionreception onon mailmail isis stoppedstopped finallyfinally thenthen thethe mailmail clientclient hashas toto bebe reconfiguredreconfigured (see(see later)later) NoNo furtherfurther configurationconfiguration changeschanges requiredrequired (e.g.(e.g. registry)registry) !! 6. Okt. techn. Seminar 8 Move from mail to imap (2)
afterafter OctoberOctober 15:15: useruser getsgets informedinformed byby emailemail thatthat hishis INBOXINBOX hashas beenbeen copiedcopied toto imapimap andand newnew emailemail isis receivedreceived onon mailmail andand imapimap useruser cancan sendsend mailmail toto stopstop mailmail receptionreception onon mailmail AfterAfter atat mostmost 4848 hourshours mailmail receptionreception onon mailmail isis stoppedstopped finallyfinally useruser hashas toto reconfigurereconfigure thethe mailmail clientclient detaileddetailed informationinformation onon thethe followingfollowing slidesslides ThereThere isis nono changechange inin thethe preferredpreferred addressaddress whenwhen sendingsending mail:mail: [email protected]@desy.de,, thethe realreal addressaddress ofof thethe INBOXINBOX inin thethe registryregistry staysstays unchangedunchanged ([email protected])([email protected])
6. Okt. techn. Seminar 9 Configuration of mail clients
mailmail clientsclients alpinealpine (successor(successor ofof pine,pine, texttext based)based) thunderbirdthunderbird mulberrymulberry (very(very powerfulpowerful,, MacOSMacOS looklook andand feel,feel, onon LinuxLinux buggy)buggy) others,others, e.g.e.g. evolution,evolution, outlook,outlook, ...... configurationconfiguration describeddescribed onon https://dvinfo.ifh.de/IMAPServerhttps://dvinfo.ifh.de/IMAPServer importantimportant parameters:parameters: serverserver namename imap.ifh.de,imap.ifh.de, protocolprotocol IMAP,IMAP, portport 143143 (TLS),(TLS), 993993 (SSL)(SSL) mailmail directorydirectory onon server:server: keepkeep emptyempty (or(or maybemaybe ~)~) correctcorrect installationinstallation ofof CACA CertificatesCertificates isis crucialcrucial forfor properproper functionality!functionality! 6. Okt. techn. Seminar 10 alpine
inin ZeuthenZeuthen alreadyalready preconfiguredpreconfigured (server(server mailmail insteadinstead ofof imap)imap) configurationconfiguration changechange toto useuse thethe newnew serverserver inbox-path={imap.ifh.de}inboxinbox-path={imap.ifh.de}inbox inin .pinerc.pinerc oror changechange InboxInbox PathPath inin alpinealpine (Setup(Setup ->-> ConfigConfig Screen)Screen) oror exportexport IMAPSERVER=imap.ifh.deIMAPSERVER=imap.ifh.de inin .zshenv.zshenv (Zeuthen(Zeuthen only)only) oror setenvsetenv IMAPSERVERIMAPSERVER imap.ifh.deimap.ifh.de inin .cshrc.cshrc (Zeuthen(Zeuthen only)only) configureconfigure alpinealpine toto displaydisplay additionaladditional foldersfolders onon serverserver setupsetup ->-> collectionListscollectionLists ->-> addadd collectioncollection arbitrary nickname, server Name: imap.ifh.de, remaining fields empty in this collection the folder junk wil get displayed new folders in this collection are visible on all mail clients talking IMAP
6. Okt. techn. Seminar 11 Reply address in alpine
settingssettings thatthat areare identicalidentical forfor allall usersusers areare writtenwritten toto thethe globalglobal alpinealpine configurationconfiguration filefile From:From: addressaddress isis differentdifferent forfor allall users,users, thethe builtbuilt inin defaultdefault willwill constructconstruct itit fromfrom domaindomain (ifh.de)(ifh.de) andand accountname.accountname. ThisThis shouldshould bebe changedchanged !! onlyonly aa problemproblem ofof (al)pine,(al)pine, otherother mailmail readersreaders willwill usuallyusually askask forfor thethe emailemail addressaddress toto bebe usedused inin thethe From:From: headerheader EvenEven worseworse onon computerscomputers notnot managedmanaged byby DESYDESY defaultdefault isis aa From:From: accordingaccording toto thethe templatetemplate [email protected]@host.domain atat DESYDESY thisthis regularlyregularly causescauses replyreply mailsmails toto bouncebounce thereforetherefore important:important: changechange From:From: inin setupsetup ->-> configureconfigure recommendedrecommended toto modifymodify asas well:well: alt-addressesalt-addresses (Alternate(Alternate Adresses)Adresses)
6. Okt. techn. Seminar 12 alpine and Multimedia Attachments
handledhandled properlyproperly (graphics,(graphics, URLURL displaydisplay inin browser,browser, sound,sound, ...)...) requiresrequires correctcorrect MIMEMIME settingssettings inin alpinealpine allall attachmentsattachments dodo havehave aa MIMEMIME typetype characterizescharacterizes typetype ofof documents,documents, e.g.e.g. Image/JPEGImage/JPEG mappingmapping ofof anan applicationapplication toto aa MIMEMIME typetype inin /etc/mailcap/etc/mailcap example: image/*; gthumb %s own rules in ~/.mailcap can enhance or replace global rules example: postscript files:application/pdf; acroread %s some programs put rules in ~/.mailcap on installation, please check!!! genericgeneric type:type: Application/OCTET-STREAMApplication/OCTET-STREAM No rule does apply, last resort: assignment of apps to file extensions mappingmapping ofof filefile extensionsextensions toto MIMEMIME typetype inin /etc/mime.types/etc/mime.types
6. Okt. own rules in .mime.types cantechn. enhance Seminar or replace global rules 13 Thunderbird Certificates
downloaddownload requiredrequired CertificateCertificate AuthorityAuthority (CA)(CA) CertificatesCertificates (see(see alsoalso https://dvinfo.ifh.de/IMAPServer#Thunderbird)) https://pki.pca.dfn.de/desy-ca/pub/cacert/g_rootcert.crthttps://pki.pca.dfn.de/desy-ca/pub/cacert/g_rootcert.crt https://pki.pca.dfn.de/desy-ca/pub/cacert/g_intermediatecacert.crthttps://pki.pca.dfn.de/desy-ca/pub/cacert/g_intermediatecacert.crt https://pki.pca.dfn.de/desy-ca/pub/cacert/g_cacert.crthttps://pki.pca.dfn.de/desy-ca/pub/cacert/g_cacert.crt inin thethe EditEdit menu:menu: Preferences:Preferences: Advanced:Advanced: ViewView CertificatesCertificates (Windows:(Windows: inin thethe ToolsTools menumenu underunder Options)Options) selectselect thethe AuthoritiesAuthorities tabtab importimport allall threethree certificatescertificates andand selectselect thethe checkboxcheckbox “use“use forfor email”email” configureconfigure otherother mailmail clientsclients accordinglyaccordingly !!!!!!
6. Okt. techn. Seminar 14 Thunderbird and Kerberos (Windows)
configureconfigure KerberosKerberos authenticationauthentication installinstall KerberosKerberos forfor WindowsWindows (kfW)(kfW) http://web.mit.edu/Kerberos/dist/http://web.mit.edu/Kerberos/dist/ startstart andand configureconfigure thethe NetworkNetwork IdentityIdentity ManagerManager realm IFH.DE, no Kerberos4 Ticket, DESY account name requestrequest ticketticket (need(need toto typetype youryour password)password) configureconfigure thunderbirdthunderbird forfor KerberosKerberos (Tools(Tools menu)menu) useuse securesecure AuthenticationAuthentication (Account(Account settings:settings: ServerServer Settings)Settings) switchswitch offoff SSPISSPI (Options:(Options: Advanced:Advanced: General:General: ConfigConfig Editor)Editor) set auth.use-sspi to false testtest iIfiIf itit works:works:
6. Okt. techn. Seminar 15 Name spaces
InformationInformation forfor experiencedexperienced usersusers
areare collectioncollection ofof rulesrules howhow mailsmails willwill bebe treatedtreated onon serverserver storagestorage format,format, accessaccess rights,rights, visibilityvisibility etc.etc. defaultdefault namename spacespace currentlycurrently storagestorage ofof mailsmails inin Maildir++Maildir++ format,format, 20102010 inin dboxdbox formatformat #mbox#mbox namename spacespace hiddenhidden fromfrom users,users, storingstoring ofof mailsmails inin mboxmbox formatformat foldersfolders cancan getget compressed,compressed, thenthen readread onlyonly wellwell suitedsuited forfor e.g.e.g. oldold sent-mailsent-mail foldersfolders
6. Okt. techn. Seminar 16 Sending of email
sendingsending ofof emailemail withoutwithout restrictionsrestrictions onlyonly withinwithin DESYDESY internalinternal serverserver isis faster,faster, nono extraextra loadload byby spammersspammers DESYDESY mailmail serversservers dodo onlyonly acceptaccept mailmail fromfrom mailmail serversservers arbitraryarbitrary machinesmachines withinwithin DESYDESY authenticatedauthenticated usersusers withwith DESYDESY cccountcccount (world(world wide)wide) authenticationauthentication againstagainst mailmail serverserver onlyonly usingusing TLSTLS (or(or SSL)SSL) byby username/passwordusername/password (always(always working,working, butbut notnot veryvery convenientconvenient)) ByBy usingusing KerberosKerberos (recommended(recommended ifif offeredoffered byby client,client, notnot Outlook)Outlook) certificatecertificate chainchain hashas toto bebe intactintact andand completecomplete cancan bebe achievedachieved byby installinginstalling thethe certificatescertificates asas describeddescribed aboveabove
6. Okt. techn. Seminar 17 Mail filtering
clientclient sideside mailmail filteringfiltering filterfilter willwill bebe workingworking withwith allall mailmail servers,servers, filteringfiltering atat clientclient startstart needsneeds toto bebe configuredconfigured separatelyseparately forfor eacheach mailmail clientclient serverserver sideside mailmail filteringfiltering filterfilter getsgets engagedengaged whenwhen emailemail isis receivedreceived eacheach mailmail clientclient doesdoes seesee thethe samesame effectseffects ofof thethe filter(s)filter(s) onon emailemail configurationconfiguration dependsdepends onon mailmail serverserver usedused procmailprocmail waswas usedused onon mail,mail, nownow onon imapimap sievesieve hashas toto bebe usedused managesievemanagesieve needsneeds toto bebe usedused toto manipulatemanipulate sievesieve scriptsscripts (usually(usually builtbuilt intointo thethe clients)clients)
6. Okt. techn. Seminar 18 Mail filtering on server
wewe havehave toto useuse aa nonnon standardstandard managesievemanagesieve configurationconfiguration portport 20092009 insteadinstead ofof 2000,2000, HostHost imap.ifh.de,imap.ifh.de, TLSTLS mustmust bebe usedused configurationconfiguration currentlycurrently onlyonly withinwithin DESYDESY (port(port blockedblocked fromfrom outside)outside) severalseveral graphicalgraphical interfacesinterfaces availableavailable thunderbird:thunderbird: sievesieve addonaddon http://sieve.mozdev.org/http://sieve.mozdev.org/ (script(script editing)editing) https://www-zeuthen.desy.de/dv-bin/imap/manage.plhttps://www-zeuthen.desy.de/dv-bin/imap/manage.pl (standalone)(standalone) https://imap.ifh.de/webmailhttps://imap.ifh.de/webmail (squirrelmail)(squirrelmail) mulberrymulberry mailmail clientclient comescomes withwith integratedintegrated interfaceinterface (IMAP(IMAP useuse ok)ok) eacheach interfaceinterface withwith separateseparate scriptscript managementmanagement mechanismsmechanisms useruser hashas toto selectselect aa singlesingle interfaceinterface OwnOwn scriptscript replacesreplaces globalglobal scriptscript (spam(spam filter)filter) unconditionallyunconditionally
6. Okt. techn. Seminar 19 Mailfilter GUI examples
6. Okt. techn. Seminar 20 Spam tagging and filtering
GlobalGlobal spamspam filterfilter installedinstalled onon imapimap cancan bebe modified,modified, replacedreplaced byby ownown filterfilter oror getget deactivateddeactivated defaultdefault filterfilter rule:rule: require "fileinto"; if header :contains "X-Spam-Level" "*****" { fileinto "junk"; } forfor spamspam filteringfiltering useuse X-Spam-Level,X-Spam-Level, notnot Subject:Subject: headerheader alsoalso validvalid foforr clientclient sideside filteringfiltering (german(german only,only, withwith pictures):pictures): http://adweb.desy.de/~gut/SpamFilterOutlook2000.htm http://dv-zeuthen.desy.de/services/mail/spamfiltereinstellungen_windows_xp/
6. Okt. techn. Seminar 21 Spam filtering using alpine
useuse globalglobal filterfilter rulesrules oror enhanceenhance defaultdefault rulerule toto movemove spamspam toto /dev/null/dev/null ifif scorescore >> 1010 useuse IndexcolorIndexcolor rulesrules e.g.e.g. “spam”“spam” rule:rule: displaydisplay aa lineline inin thethe indexindex inin grey,grey, ifif scorescore >> 00 selectselect possiblepossible spamspam byby enteringentering ;; rr spamspam
6. Okt. techn. Seminar 22 Frequently asked questions
maximalmaximal allowedallowed sizesize ofof emailsemails toto sendsend allall centralcentral DESYDESY mailmail serversservers dodo acceptaccept upup toto 50MB50MB mailsmails binarybinary filesfiles << 3030 MBMB onlyonly (get(get encoded,encoded, growgrow becausebecause ofof that)that) acceptedaccepted maximummaximum sizesize onon manymany otherother machinesmachines 10MB10MB IfIf anan emailemail exceedsexceeds thethe sizesize limit,limit, thethe sendingsending useruser doesdoes getget aa bouncebounce withwith thethe namename ofof thethe complainingcomplaining mailmail server,server, pleaseplease readread carefullycarefully betterbetter option:option: putput filefile inin ~/public/www/
6. Okt. techn. Seminar 23 Frequently asked questions (2)
whywhy obviousobvious SPAMSPAM mailsmails withwith aa regularregular patternpattern areare notnot taggedtagged ourour firstfirst aimaim isis toto minimizeminimize thethe amountamount ofof goodgood emailsemails inin spam,spam, notnot minimizingminimizing thethe numbernumber ofof spamspam emailsemails inin thethe INBOXINBOX ThereThere isis nono adaptionadaption ofof spamspam rulesrules toto thethe patternspatterns seenseen atat DESY.DESY. ThisThis wouldwould causecause extraextra workwork andand couldcould badlybadly influenceinfluence thethe scoringscoring rulesrules SeeminglySeemingly efficientefficient rulesrules willwill workwork onlyonly forfor aa fewfew weeksweeks beforebefore spammersspammers useuse newnew methodsmethods (example:(example: spamspam alsals JPEGJPEG picture)picture) II dodo getget spamspam emailemail withwith mymy addressaddress inin thethe From:From: headerheader allall sendersender andand recipientrecipient addressesaddresses cancan easilyeasily bebe spoofedspoofed onlyonly thethe addressaddress onon thethe envelopeenvelope hashas toto bebe correct,correct, envelopeenvelope informationinformation isis nevernever displayed.displayed. TheThe emailemail seenseen correspondscorresponds toto thethe contentcontent ofof anan ordinaryordinary mail,mail, infoinfo therethere cancan bebe differentdifferent fromfrom envelopeenvelope
6. Okt. techn. Seminar 24 Frequently asked questions (3)
IsIs aa givengiven emailemail spam?spam? visiblevisible From:From: andand To:To: headerheader easilyeasily spoofablespoofable downloaddownload ofof picturespictures cancan yieldyield informationinformation aboutabout youryour computercomputer VisibleVisible linkslinks (URLs)(URLs) frequentlyfrequently pointpoint toto spammerspammer sitessites inspectinginspecting allall headersheaders helpshelps Received: from dhcp-077-211-218-116.chello.nl (user-5433e1d5.lns6-c13.telh.dsl.pol.co.uk...) carefullycarefully lookinglooking atat URL'sURL's helpshelps 6. Okt. techn. Seminar 25 What is next? untiluntil endend ofof OctoberOctober movemove ofof allall ZeuthenZeuthen INBOXesINBOXes toto imapimap spamassassinspamassassin upgradeupgrade ffoorr improvedimproved spamspam taggingtagging newnew versionversion 3.33.3 shouldshould bebe readyready soonsoon beginningbeginning 20102010 newnew moremore efficientefficient formatformat forfor storingstoring mailmail importantimportant ffoorr backup,backup, fewfew largelarge filesfiles insteadinstead ofof manymany smallsmall onesones getsgets implementedimplemented inin dovecotdovecot 2.02.0 improvingimproving documentation,documentation, optimizingoptimizing thethe mailmail serverserver andand clientclient configurationconfiguration (feedback(feedback fromfrom usersusers welcome)welcome) UNIXUNIX mailmail storestore inin HamburgHamburg (mail.desy.de)(mail.desy.de) willwill bebe basedbased onon dovecotdovecot asas wellwell (currently(currently inin testingtesting phase)phase) 6. Okt. techn. Seminar 26 Useful links DESYDESY specificspecific linkslinks http://dv-zeuthen.desy.de/services/mail/ (general info) https://dvinfo.ifh.de/IMAPServer (general info for IMAP server) https://dvinfo.ifh.de/MailReaderConfiguration (mail configuration) https://dvinfo.ifh.de/MailFilter (general info on mail filtering) https://imap.ifh.de/webmail (squirrelmail for configuring own mail filter) https://www-zeuthen.desy.de/dv-bin/imap/manage.pl (dito) https://pki.pca.dfn.de/desy-ca/pub/ (Certificates for DESY) GeneralGeneral linkslinks http://wiki.dovecot.org/ (dovecot wiki) http://sieve.info/ (sieve filter resources) http://pigeonhole.dovecot.org/ (currently used sieve implementation) 6. Okt. techn. Seminar 27 Questions and comments ? dovecotdovecot Webster:Webster: Main Entry: dove·cote Pronunciation: \'də v-,kōt, -,kät\ Variant(s): also dove·cot \-,kät\ Function: noun Date: 15th century 1 : a small compartmented raised house or box for domestic pigeons 2 : a settled or harmonious group or organization 6. Okt. techn. Seminar 28