DOCSLIB.ORG

How Emails Are Sent from Xero Technical Discussion

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
How Emails Are Sent from Xero Technical Discussion How emails are sent from Xero Technical discussion In June 2013 we made a change to the way emails are sent from Xero. Some of our users have asked us why the change was necessary and whether we are planning to provide more options in the future. Be warned ­ this is a fairly technical topic so take a deep breath. Xero allows organisations to raise invoices and statements and send them to their customers. Sending email on behalf of hundreds of thousands of Xero organisations, to the millions of their customers is a significant undertaking, and one that we take seriously. Recently we changed the mechanism by which we send customer invoice emails. This post is an attempt to describe the new setup, the reasoning behind the decision, and a road­map for future changes that we hope to make. Setting the scene: The email landscape has changed dramatically over the past 5 years, and mail servers and email ISPs have all developed unique practices to try and shield their users from spam and other unsolicited email that currently floods the internet. Over the past 10 years, several standards and de­facto practices have evolved to help mail servers make decisions on which emails to trust: ● Sender Policy Framework (SPF) ­ a 10 year old standard that lets the owner of a mail domain (e.g. gmail.com) specify which mail servers are allowed to send email ● DomainKeys Identified Mail (DKIM) ­ a slightly newer standard that allows the sender of an email to digitally sign their email to prove that it came from them ● Grey listing ­ a heuristic that some receiving mail servers use to slow down the receipt of email, on the theory that spammers won’t attempt to queue and retry messages ● Spam filtering ­ a set of algorithms that receiving mails servers can run on email to try and determine if the content of the email is “spammy” ● Blacklisting ­ various databases of IP addresses or email addresses that are known to send spam ● Domain­based Message Authentication, Reporting & Conformance (DMARC) ­ a recent specification that combines SPF and DKIM to allow email senders to prove that the mail was sent by them We realise that our users absolutely, positively want email that they send to be received by their customers ­­ their cashflow and business depend on timely payments. Even though email is by design an unreliable communication mechanism, we are trying to make it as robust and reliable as possible for Xero customers to use email to send invoices to their customers. Our previous approach: Our previous approach to sending invoice emails from within Xero was to “impersonate” or send email “on behalf” of the user currently logged in to Xero. e.g. If [email protected] had signed up to Xero, and proved that they controlled the email address, then we would happily send out email that looked like this: From: Joe Bloggs <[email protected]> To: A Customer <[email protected]> Subject: Here is your invoice, pay me now! In 2006, this was a fairly reliable solution, and most mail servers would happily accept these emails and deliver them to the recipient. In the subsequent years, it was possible for Joe Bloggs to add an SPF record to their flatnotfound.com to indicate that Xero was able to send email on their behalf. This worked reasonably well for a few more years. The Problem: It is now no longer possible for SaaS vendors to reliably send email “on behalf” of their customers. If a recipient mail server receives an email like the above example, the mail server will perform some checks to try and decide whether this email was legitimately sent by Joe Bloggs. One of the indications the server will use is the IP address of the server that delivered the message to it. If the server does not look like a legitimate server for Joe Bloggs’ ISP (flatnotfound.com), then that’s a negative mark against the email. Traditionally a combination of reverse DNS lookups and SPF would be used to perform this check. These days, setting SPF records is not sufficient to prove the authenticity of an email, and this example email illustrates the reason why: Envelope Sender: [email protected] From: Joe Bloggs <[email protected]> To: A Customer <[email protected]> Subject: Here is your invoice, pay me now! The “Envelope Sender” address is never displayed to regular email users, but it is actually the address that SPF uses to test the authenticity of an email. In this example, if this email message arrived from a server that was legitimately run by “evilspammer.net”, then technically that’s all that is needed to be legitimate from an SPF perspective ­­ even though the email displayed in the recipient’s mail client doesn’t indicate it comes from evilspammer.net! For this reason, SPF is no longer the solution to sending email “on behalf” of customers like it was when we launched Xero a few years ago. We have had this confirmed by email experts from Facebook, Yahoo! and JPMorgan Chase when we spoke to them in February this year. In fact, they confirmed what we were seeing, which is that there’s no way to reliably send email while pretending to be someone else. The previous method of sending Xero emails out by impersonating the Xero user was untenable, and we would see continuing increases in the numbers of emails that would fail to deliver. Problems with SPF: We’ve already discussed how SPF on it’s own is not sufficient, and that it relies on the verification of a part of the email message that’s not actually displayed to an end user. Our invoice emails actually looked roughly like this: Envelope Sender: [email protected] From: Joe Bloggs <[email protected]> To: A Customer <[email protected]> Subject: Here is your invoice, pay me now! For this email to pass an SPF test, the IP address of the mail server is compared to the SPF records of the “Envelope Sender” domain of notifications.xero.com. Xero has control of these records, and so we were able to make sure that the SPF record for notifications.xero.com was always valid. Some mail servers would also run SPF­like tests against the visible “From” address that displays in the email, even though that’s not strictly part of the SPF standard, effectively asking the question: “Is Xero’s email server allowed to send email from flatnotfound.com?” In order to pass this test, every Xero customer would need to create an SPF record in their DNS to allow Xero’s mail servers to send email on their behalf. For some Xero customers that are IT­savvy, and in control of their own DNS, this was indeed possible, and over the past few years approximately one hundred Xero customers have managed to do this successfully. However, for most Xero customers this is infeasible: ● Many Xero customers use an “ISP” email address, rather than their own domain name (e.g. [email protected] rather than [email protected]). This means that they have no control over the configuration of their ISP. ● Customers that had their own domain name (e.g. MyCompany.com) but weren’t able to change DNS, had a large company with strict IT policies, or were part of a multi­national. These customers aren’t easily able to create SPF records. ● Customers that use SPF, but didn’t include Xero’s servers in their SPF records may be hard­failing all impersonated mail from other sources. Our analysis showed this was potentially the case for about 7% of customers. ● Corporate customer’s mail servers would often reject Xero invoice emails that they sent to themselves, even if SPF was correctly set up. E.g. a corporate mail server for MyCompany.com would say “I’ve recieved mail from xero.com’s mail server’s that is pretending to be from MyCompany.com. I know that I’m the only mail server for MyCompany.com, so this is obviously spam” and throw it away. ● Creating valid SPF records is actually hard. One only needs to look around the internet and on our own Community to see that people don’t understand the nuances of SPF, and a simple mistake in configuration can stop delivery of all your company’s email. This goes against our mantra of being the “World’s easiest accounting system”. The (2013) solution: We absolutely want to deliver email for our customers, to their customers. Asking our users to download invoices and send them from their desktop is cumbersome and inefficient ­ although we realise for some customers that it will always be their preferred approach so that they have complete control. Candidate solutions: ● Continue impersonating email ○ Unreliable, not recommended by experts. ○ Requires every Xero user to configure SPF correctly, even though that won’t guarantee delivery. ● Send email via our customer’s email servers ○ Technically infeasible (we’d have to collect login credentials, connect to hundreds of thousands of authenticated mail servers to send email, and deal with the myriad of different ways that could fail). ○ We’d be outsourcing the “email delivery” issues to our customers, rather than being responsible for delivering mail correctly on their behalf. ● Deliver email with a .xero.com “From” address ○ We can guarantee that messages are correctly authenticated. ○ We control the full infrastructure for the email sending. ○ We can track deliverability and bounces. Ultimately, we made the decision to choose the third option, and change the way that we deliver invoice email.
Load more

Recommended publications
  • DMARC — Defeating E-Mail Abuse
    CERT-EU Security Whitepaper 17-001 DMARC — Defeating E-Mail Abuse Christos Koutroumpas ver. 1.3 February 9, 2017 TLP: WHITE 1 Preface E-mail is one of the most valuable and broadly used means of communication and most orga- nizations strongly depend on it. The Simple Mail Transport Protocol (SMTP) – the Internet’s underlying email protocol – was adopted in the eighties and is still in use after 35 years. When it was designed, the need for security was not so obvious, and therefore security was not incor- porated in the design of this protocol. As a result, the protocol is susceptible to a wide range of attacks. Spear-phishing campaigns in particular can be more successful by spoofing (altering) the originator e-mail address to imper- sonate a trusted or trustworthy organization or person. This can lead to luring the recipient into giving away credentials or infecting his/her computer by executing malware delivered through the e-mail. While raising user awareness on how to avoid e-mail fraud is recommended, the Verizon Data Breach Investigations Report indicates that more needs to be done. The DBIR report reveals that 30% of all phishing e-mail messages were opened by the recipients and with 12% clicked on the content and executed malicious code. The median time for the first user of a phishing campaign to open the malicious email is 1 minute, 40 seconds. The median time to the first click on the attachment was 3 minutes, 45 seconds. These statistics highlight the risk for an organization on the receiving end of spear-phishing e-mails.
    [Show full text]
  • Delivering Results to the Inbox Sailthru’S 2020 Playbook on Deliverability, Why It’S Imperative and How It Drives Business Results Introduction to Deliverability
    Delivering Results to the Inbox Sailthru’s 2020 Playbook on Deliverability, Why It’s Imperative and How It Drives Business Results Introduction to Deliverability Every day, people receive more than 293 billion Deliverability is the unsung hero of email marketing, emails, a staggering number that only represents ultimately ensuring a company’s emails reach their the tip of the iceberg. Why? The actual number intended recipients. It’s determined by a host of of emails sent is closer to 5.9 quadrillion, with the factors, including the engagement of your subscribers overwhelming majority blocked outright or delivered and the quality of your lists. All together, these factors to the spam folder. result in your sender reputation score, which is used to determine how the ISPs treat your email stream. Something many people don’t realize is that to the Deliverability is also a background player, so far in the major Internet Service Providers (ISPs) — Gmail, shadows that many people don’t think about it, until Yahoo!, Hotmail, Comcast and AOL — “spam” there’s a major issue. doesn’t refer to marketing messages people may find annoying, but rather malicious email filled with That’s why Sailthru’s deliverability team created this scams and viruses. In order to protect their networks guide. Read on to learn more about how deliverability and their customers, the ISPs cast a wide net. If a works on the back-end and how it impacts revenue, message is deemed to be spam by the ISP’s filters, it’s your sender reputation and how to maintain a good dead on arrival, never to see the light of the inbox, as one, and best practices for list management, email protecting users’ inboxes is the top priority of any ISP.
    [Show full text]
  • Presentations Made by Senders
    SES ���� ��� � �� � � � � � � � ������������� DomainKeys ��������� SPF ��������������������� ���������� ����������������� ������������������������������������������������ Contents Introduction 3 Deployment: For Email Receivers 6 Audience 3 Two Sides of the Coin 6 How to Read this White Paper 3 Recording Trusted Senders Who Passed Authentication 6 A Vision for Spam-Free Email 4 Whitelisting Incoming Forwarders 6 The Problem of Abuse 4 What To Do About Forgeries 6 The Underlying Concept 4 Deployment: For ISPs and Enterprises 7 Drivers; or, Who’s Buying It 4 Complementary considerations for ISPs 7 Vision Walkthrough 5 Deployment: For MTA vendors 8 About Sender Authentication 8 Which specification? 8 An Example 8 Conformance testing 8 History 8 Perform SRS and prepend headers when forwarding 8 How IP-based Authentication Works 9 Add ESMTP support for Submitter 8 The SPF record 9 Record authentication and policy results in the headers 8 How SPF Classic Works 9 Join the developers mailing list 8 How Sender ID works 9 Deployment: For MUA vendors 9 How Cryptographic Techniques Work 0 Displaying Authentication-Results 9 Using Multiple Approaches Automatic switching to port 587 9 Reputation Systems Deployment: For ESPs 20 Deployment: For Email Senders 2 Don’t look like a phisher! 20 First, prepare. 2 Delegation 20 Audit Your Outbound Mailstreams 2 Publish Appropriately 20 Construct the record 2 Deployment: For Spammers 2 Think briefly about PRA and Mail-From contexts. 3 Two Types of Spammers 2 Test the record, part 3 Publish SPF and sign with DomainKeys. 2 Put the record in DNS 3 Stop forging random domains. 2 Test the record, part 2 4 Buy your own domains. 2 Keep Track of Violations 4 Reuse an expired domain.
    [Show full text]
  • Combatting Spam Using Mimedefang, Spamassassin and Perl
    Combating Spam Using SpamAssassin, MIMEDefang and Perl Copyright 2003 David F. Skoll Roaring Penguin Software Inc. (Booth #23) Administrivia Please turn off or silence cell phones, pagers, Blackberry devices, etc... After the tutorial, please be sure to fill out an evaluation form and return it to the USENIX folks. 2 Overview After this tutorial, you will: Understand how central mail filtering works. Know how to use MIMEDefang to filter mail. Be able to integrate SpamAssassin into your mail filter. Know how to implement mail filtering policies with MIMEDefang and Perl. Know how to fight common spammer tactics. 3 Outline Introduction to Mail Filtering Sendmail's Milter API MIMEDefang Introduction, Architecture Writing MIMEDefang Filters SpamAssassin Integration Advanced Filter Writing Fighting Common Spammer Tactics Advanced Topics Policy Suggestions 4 Assumptions I assume that you: Are familiar with Sendmail configuration. You don't need to be a sendmail.cf guru, but should know the basics. Are familiar with Perl. Again, you don't need to be able to write an AI program in a Perl one- liner, but should be able to read simple Perl scripts. Are running the latest version of Sendmail 8.12 on a modern UNIX or UNIX-like system. 5 Why Filter Mail? The old reason: to stop viruses. The new reason: to stop spam and inappropriate content. Blocking viruses is easy. Block .exe and similar files, and test against signature databases. Blocking spam is hard, but becoming increasingly important. Organizations can even face lawsuits over inappropriate content. 6 Mail filtering is required for many reasons. In addition to the reasons given on the slide, you might need to filter outgoing mail as well to prevent virus propagation, dissemination of sensitive information, etc.
    [Show full text]
  • Account Administrator's Guide
    ePrism Email Security Account Administrator’s Guide - V10.4 4225 Executive Sq, Ste 1600 Give us a call: Send us an email: For more info, visit us at: La Jolla, CA 92037-1487 1-800-782-3762 [email protected] www.edgewave.com © 2001—2016 EdgeWave. All rights reserved. The EdgeWave logo is a trademark of EdgeWave Inc. All other trademarks and registered trademarks are hereby acknowledged. Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other product and company names mentioned herein may be the trademarks of their respective owners. The Email Security software and its documentation are copyrighted materials. Law prohibits making unauthorized copies. No part of this software or documentation may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into another language without prior permission of EdgeWave. 10.4 Contents Chapter 1 Overview 1 Overview of Services 1 Email Filtering (EMF) 2 Archive 3 Continuity 3 Encryption 4 Data Loss Protection (DLP) 4 Personal Health Information 4 Personal Financial Information 5 Document Conventions 6 Other Conventions 6 Supported Browsers 7 Reporting Spam to EdgeWave 7 Contacting Us 7 Additional Resources 7 Chapter 2 Portal Overview 8 Navigation Tree 9 Work Area 10 Navigation Icons 10 Getting Started 11 Logging into the portal for the first time 11 Logging into the portal after registration 12 Changing Your Personal Information 12 Configuring Accounts 12 Chapter 3 EdgeWave Administrator
    [Show full text]
  • Set up Mail Server Documentation 1.0
    Set Up Mail Server Documentation 1.0 Nosy 2014 01 23 Contents 1 1 1.1......................................................1 1.2......................................................2 2 11 3 13 3.1...................................................... 13 3.2...................................................... 13 3.3...................................................... 13 4 15 5 17 5.1...................................................... 17 5.2...................................................... 17 5.3...................................................... 17 5.4...................................................... 18 6 19 6.1...................................................... 19 6.2...................................................... 28 6.3...................................................... 32 6.4 Webmail................................................. 36 6.5...................................................... 37 6.6...................................................... 38 7 39 7.1...................................................... 39 7.2 SQL.................................................... 41 8 43 8.1...................................................... 43 8.2 strategy.................................................. 43 8.3...................................................... 44 8.4...................................................... 45 8.5...................................................... 45 8.6 Telnet................................................... 46 8.7 Can postfix receive?..........................................
    [Show full text]
  • Email Sender Authentication Development and Deployment
    EMAIL SENDER AUTHENTICATION DEVELOPMENT AND DEPLOYMENT (PROJECT CHEESEPLATE) Volume I Technical and Management Proposal pobox.com IC Group, Inc. [email protected] v1.01 20041217 Full Proposal Control Number EB8A Email Sender Authentication OFFICIAL TRANSMITTAL LETTER IC Group, Inc., a New York State corporation, doing business as pobox.com, respectfully submits a proposal in response to solicitation BAA04-17 for Cyber Security Research and Development. It is submitted under Category 3, Technical Topic Area 7, Technologies to Defend Against Identity Theft, for consideration as a Type II Prototype Technology. Solicitation Title: BAA 04-17 Topic Title: Technologies to Defend Against Identity Theft Type Title: Type II (Prototype Technologies) Full Proposal Control Number: EB8A Proposal Title: Email Sender Authentication A companion proposal, Reputation System Clearinghouse (1RGT), is also being submitted under the same category and type. We request that these two proposals be read together. This proposal should be read first. This proposal was authored by Meng Weng Wong, Founder and Chief Technology Officer for Special Projects. He can be contacted at [email protected]. Meng Weng Wong IC Group, Inc. 1100 Vine St Ste C8 Philadelphia, PA 19107 December 15th 2004 EIN: 113236046 Central Contractor Registration: 3EKUCT Email Sender Authentication 2 EXECUTIVE SUMMARY Pobox.com aims to fight phishing by adding sender authentication “Phishing” is a class of high-tech scam that functionality to the Internet email system. First we will build a library uses fraudulent e-mail to deceive consum- ers into visiting fake replicas of familiar to implement a useful set of recently devised anti-forgery specifica- Web sites and disclosing their credit card tions, including ip-based approaches such as SPF and crypto-based numbers, bank account information, Social approaches such as DomainKeys.
    [Show full text]
  • How to Make Sure Your Emails Land in Your Prospects' Inboxes
    How to Make Sure Your Emails Land in Your Prospects’ Inboxes How to Make Sure Your Emails Land in Your Prospects’ Inboxes | 1 Table of Contents Why Aren’t My Emails Getting Through? 3 Authentication 5 Permissions 7 Reputation 9 Sender Reputation 9 Cleanliness & Monitoring 10 List Source 10 New Domain Address Warming 11 Review Bounces 14 Monitoring Blacklists 15 Follow the Rules - CAN-SPAM, CASL, GDPR & CCPA 16 Engagement 17 Know Your Audience - Relevance, Frequency & Content Review 17 Unsubscribe Links 18 Conclusion 19 How to Make Sure Your Emails Land in Your Prospects’ Inboxes | 2 Why would my emails never make it to their destination? As we move further into the era of technology, email has become the primary source of professional communication. As a result, bad actors are doing whatever it takes to get you to view their emails. How many members of the Nigerian royal family have contacted you to transfer their fortune to your bank account? In response to more frequent attempts to phish, hack, and send spam, Internet Service Providers (ISPs) are doing everything they can to protect their customers from potentially unsolicited email, ! including blocking bulk email sends from new domains and internet protocol (IP) addresses. It’s important to remember that ISPs are always looking to protect their users (and investors). Because you’re sending emails to reach out to your prospects, these new measures have a direct impact on your ability to have your legitimate emails delivered to the inbox. How to Make Sure Your Emails Land in Your Prospects’ Inboxes | 3 Ultimately it is the practices of your company, and your engagement strategy, that determines whether or not your messages get through.
    [Show full text]
  • Guide Deliverability.Pdf
    email delivery for IT professionals INTRODUCTION................................................................................................................................................ 3 HOSTING AND HARDWARE ............................................................................................................................... 4 IP/DNS ............................................................................................................................................................. 5 DNS Naming................................................................................................................................................ 5 I already have a domain.......................................................................................................................... 5 I'm sending on behalf of someone else................................................................................................... 5 I'm sending on behalf of several companies............................................................................................ 6 IP Ranges .................................................................................................................................................... 6 Test IPs ....................................................................................................................................................... 6 Registrars...................................................................................................................................................
    [Show full text]
  • Adoption of Email Anti-Spoofing Schemes: a Large Scale Analysis
    This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSM.2021.3065422, IEEE Transactions on Network and Service Management IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT 1 Adoption of Email Anti-Spoofing Schemes: A Large Scale Analysis Sourena Maroofi, Maciej Korczynski,´ Arnold Hölzel, Andrzej Duda, Member, IEEE Abstract—Sending forged emails by taking advantage of a real Microsoft login page to steal user credentials [2]. In this domain spoofing is a common technique used by attackers. paper, we investigate the second type of email spoofing. The lack of appropriate email anti-spoofing schemes or their The Simple Mail Transfer Protocol (SMTP) for email dis- misconfiguration may lead to successful phishing attacks or spam dissemination. In this paper, we evaluate the extent of tribution does not provide support for preventing spoofing [3] the SPF and DMARC deployment in two large-scale campaigns so mail systems need to rely on security extensions such as measuring their global adoption rate with a scan of 236 million the Sender Policy Framework (SPF) [4], the DomainKeys domains and high-profile domains of 139 countries. We propose Identified Mail (DKIM) [5], and Domain-based Message Au- a new algorithm for identifying defensively registered domains thentication, Reporting, and Conformance (DMARC) [6] to and enumerating the domains with misconfigured SPF rules by emulating the SPF check_function. We define for the first time authenticate the sender and decide what to do with suspicious new threat models involving subdomain spoofing and present emails.
    [Show full text]
  • Administrator's Guide
    Kerio Connect Administrator’s Guide Kerio Technologies 2011 Kerio Technologies s.r.o. All rights reserved. This guide provides detailed description on Kerio Connect, version 7.2. All additional modifications and updates reserved. For current versions of the product and related manuals, check http://www.kerio.com/connect/download/. Information regarding registered trademarks and trademarks are provided in appendix A. Contents 1 Introduction .................................................................. 10 1.1 Additional documentation ............................................... 10 1.2 Quick Checklist .......................................................... 10 2 Installation .................................................................... 13 2.1 System requirements .................................................... 13 2.2 Conflicting software ..................................................... 14 2.3 Firewall configuration .................................................... 14 2.4 Installation .............................................................. 15 2.5 Configuration Wizard .................................................... 23 2.6 Upgrade and Uninstallation .............................................. 25 3 Kerio Connect components .................................................... 28 3.1 Kerio Connect Monitor ................................................... 28 3.2 Standalone processes of the server ....................................... 31 4 Kerio Connect administration ................................................
    [Show full text]
  • Email Authentication Via Domainkeys Identified Mail (DKIM)
    IronPort Email Authentication W H I T E P A P ER Executive Summary The problems of spam, viruses, phishing and most email denial-of-service attacks can all be traced back to a single common cause – lack of authentication in the email protocol SMTP. TABLE OF CONTENTS 1 Executive Summary This lack of authentication means that a receiving mail server cannot reliably 2 Definitions verify that a particular message is in fact from the sender it purports to be from, making it harder to identify friend from foe. 2 History 3 The Authentication Problem The industry has recognized this shortcoming, and a great deal of effort 4 Sender ID and DomainKeys has been put into developing a new standard that will “overlay” SMTP Identified Mail and provide the sender authentication that is so desperately needed. This 9 Adoption Status paper will present a brief history of how this problem evolved, explore the pluses and minuses of the leading standards proposals, and highlight some 10 Why Authenticate? recommendations. 11 The Solution To Bounce Attacks 11 IronPort Systems’ Adoption Recommendations 12 Appendix D O C R E V 0 2 . 0 8 1 IRONPORT EMAIL AUTHENTICATION WHITE PAPER DEFINITIONS Email nomenclature can be a bit confusing, so it is useful to start with some definitions. An email message has an addressing scheme similar to a postal message: HELO/EHLO: The initial contact command between a sending and a receiving mail server, indicating an SMTP conversation. Envelope sender: The address of the sending mail server; not exposed to the end-user, used for managing bounces.
    [Show full text]