Enterprise OSS Collaboration: RHEV3, RHEL6, Zimbra and iSCSI
Will Foster Sr Sysadmin and IT Storage Lead, Red Hat February 18, 2013
Will Foster, Red Hat Inc. This is a talk on..
● Architectural reference for how Red Hat IT has deployed Zimbra for collaboration services ● Design and layout ● Usage, footprint,
● Inherent benefits of the design ● Config management/provisioning ● Modularity ● Storage snapshot-based recovery
● Future enhancements in automation/admin ● 'SnapCreator' framework
Will Foster, Red Hat Inc. This is not a talk on..
● Migration guide from Exchange → Zimbra
● A setup guide or sales demo for Zimbra
● Informed discussion about licensing
● Anything specific to Red Hat internal services
● A guide for bathtub plutonium enrichment
Will Foster, Red Hat Inc. Evolution of Email
Will Foster, Red Hat Inc. Evolution of Email
● 1971 – First ARPANET email was set
● 1971 – Mailbox Protocol
● 1980+ SMTP protocol: start of wide adoption ● 1983+ Sendmail and lots of beards starting to grow..
● 1987-1991 – Microsoft and MAPI ● Proprietary protocol ● Spec's finally released in 2007
● 2001: 42% of all public mail servers are Sendmail ● 2012: 12% of all public mail servers are Sendmail ● 2012: 43% of all public mail servers are Exim ● 2012: 23% of all public mail servers are Postfix ● 2012: 12% of all public mail servers are Microsoft Exchange
Source: E-soft survey: http://tinyurl.com/avykuk3
Will Foster, Red Hat Inc. Need for Centralized Mail Services
● Legal discovery/retention
● Widespread use of calendaring
● Administration overhead in rapid growth
● Decentralized/departmental email servers still popular in academia
Will Foster, Red Hat Inc. Why Zimbra?
● Need for 'enterprise' collaboration
● Activesync/mobility (Android, IOS)
● Exchange/Notes/etc compatible Calendaring
● OSS stack components
● Avoid lock-in
● Multiple access methods and choice of client
– IMAP, POP, web interface, external webmail, pushmail
Fun Tip #2343: The name “Zimbra” was derived from the Talking Heads 1979 album “Fear of Music”
Will Foster, Red Hat Inc. Why use it?
● Stack built on OSS technologies
● Separate MX infrastructure
● Modular vs. Monolithic
– Isolated vs. Glorious Failure
● Postfix, MySQL, Nginx, Jetty Apache, OpenLDAP, memcached
● Less Evil..
● "All mail clients suck. This one just sucks less." - Mutt (1995)
Will Foster, Red Hat Inc. Vendor Lock-in and risks of proprietary standards
Microsoft Exchange 5.5 → 2000/2003 ● Requirement of full Active Directory infrastructure ● “This is my cousin, Frank. He can crash here right?”
Client Access: MAPI (limited to MS Outlook variants or typically unsupported plugins for MAPI (Evolution) ● IMAP4/POP3 can be enabled but not default
Mailbox Format: proprietary ● However some vendors have migration utilities (google apps, Zimbra, Zarafa, others)
Will Foster, Red Hat Inc. Zimbra Software Components
Will Foster, Red Hat Inc. Initial Zimbra Architecture (2008-2010)
Cisco MDS FC SAN RHEL4/5 Bare Metal Red Hat Cluster Suite on Netapp
Components. Experience.
RHEL bare-metal IBM blades Zimbra/RHCS only supported on EL4 at the time Red Hat Cluster Suite DM-Multipath experiences on EL4 could be better Zimbra 6.x (network edition) Bare-metal & clustered setup takes longer to provision/scale 4Gb Fibrechannel SAN LUNs
RHEL5.x Sendmail MX tier Nginx/Apache Proxy Tier
OpenLDAP Backend
Will Foster, Red Hat Inc. Zimbra Architecture Overview Today
10GbE iSCSI Storage RHEV3.x RHEL6 Mail Nodes (Netapp)
Components. Benefits.
RHEV3.0 Hypervisors Modular Design RHEL6.3 Mail and MTA Nodes Fast provisioning/deployment Zimbra 7.2.0 (network edition) Snapshot based backups Netapp 10GbE iSCSI LUNs OSS stack components RHEL6.3 Sendmail MX tier Nginx/Apache Proxy Tier
OpenLDAP Backend
Will Foster, Red Hat Inc. Zimbra Architecture: Application Overview
● 13 Mailbox nodes
● 1 Failover/reserved mailbox node ● Restores, reserved capacity, etc
● 2 Utility Nodes (1 x archive , 1 x admin)
● 2 Proxy Nodes (Apache/Nginx)
● 3 LDAP nodes (2 x replica, 1 x master)
● 3 MTA nodes (load balanced)
● 5 Internal MX servers (sendmail)
Will Foster, Red Hat Inc. Zimbra Architecture: Usage and Footprint
● 7,500 User accounts ● Includes team calendars, shared accounts, meeting rooms ● 300 mailboxes per node on average
● 1,216,058 inbound mails handled per work day ● Average 965,388 inbound mails per day
● 41TB iSCSI LUN mailbox storage
● 37TB iSCSI off-site replication
● 49TB in NFS local mailbox backups
Will Foster, Red Hat Inc. Zimbra Architecture: Application Overview
Will Foster, Red Hat Inc. Zimbra Architecture: Mail Flow Overview
Will Foster, Red Hat Inc. Zimbra Architecture: Backend Storage & DR
Will Foster, Red Hat Inc. Why iSCSI ?
● Operationally cheap SAN block storage
● Great pricepoint (no need for expensive $$ FC SAN)
● First moved to RHEV at version 2.2: no direct FC SAN support
● Introduction of 10GbE Networking into Red Hat Infrastructure
● Transactional workloads, Lots of small files = better on block
● Performance onpar with FC: TOE, Jumbo frames, 10GbE
Will Foster, Red Hat Inc. Netapp Backend Technology
● 3 x HA-pairs of FAS3270A (primary) - Mail nodes/resources spread evenly across HA pairs
● 1 x HA pair of FAS3270A (off-site)
● 10GbE iSCSI LUN connectivity: Cisco Nexus 5k
● Zimbra Mailbox deduplication average: 27%
● Zimbra off-site DR accomplished by 'Flexclones' - instant, innode pointer based volumes created from snapshot
Will Foster, Red Hat Inc. Spam and Filtering
● Milter based application (MimeDefang) ● Scans email and assigns spam score before we taking responsibility for message. Do not have to “bounce” or “discard” spam messages ● Calls Anti-Virus and Spam Scoring apps during the SMTP dialog
● Subscription based RBLs (commercial) ● SpamHaus ● Pattern matching in message content ● SpamAssassin for scoring
Will Foster, Red Hat Inc. Automation and Provisioning Concepts
● Cobbler/Puppet help automate VM provisioning and configs
● Keep nothing important on VM's – always use shared storage
● Virtual machines/servers are cheap commodities more easily replaced than fixed in most cases
● Quick rate of provision/deploy = faster scale
● ISCSI storage = one command away from block storage
Will Foster, Red Hat Inc. Future Improvements: Snap Creator
● Snap creator is an open framework that integrates with Netapp snapshot commands and 3rd party applications
● “Snap Creator for Zimbra” allows full and incremental snapshot based backups of multi-node Zimbra environments while online.
● Application and Crash consistent, Application-aware
● OSS software supported by Netapp
● Our future improvement: automated snapshot backups
Will Foster, Red Hat Inc. Email Musings
● More than 97% of all emails sent over the net are unwanted
● Re: Re: Re: Re: Re: Re: Kittens ← most popular subjects
● The first known spam email, advertising a DEC product presentation, was sent in 1978 by Gary Thuerk
● Avoid HTML email when possible, it can get out of hand.. (if you use HTML email make sure to use
Will Foster, Red Hat Inc. How I view HTML email being taken too far
Will Foster, Red Hat Inc. Summary
● Proprietary solutions can cause you lock-in and headache
● Build solutions which are modular, avoid monolithic design
● Aim for ease and automation in deployment
● Block-level storage tends to work better with small files and transactional workloads (iSCSI, SAN)
● Many (mostly) OSS collaboration solutions out there.. Zarafa, Zimbra, Atmail, etc.
● Every organization is different
Will Foster, Red Hat Inc. References, Attributions and Usage
● http://riser38.deviantart.com/art/Clone-Army-348790671
● https://en.wikipedia.org/wiki/Microsoft_Exchange_Server
● http://wikipedia.org (multiple)
● http://ko.fotopedia.com/items/flickr-3189975360
● (cc) JD Lasica/Socialmedia.biz
● http://www.securityspace.com/s_survey/data/man.201112/mxsurvey.html
● http://singletrackworld.com/forum/forum/off-topic
● https://www.zimbra.com/docs/os/6.0.8/administration_guide/2_Overview%20System%20Architecture.03.4.html
● https://en.wikipedia.org/wiki/Email_spam#Statistics_and_estimates
● http://www.netapp.com/us/products/management-software/snapcreator-framework.aspx
Will Foster, Red Hat Inc.