How are sent from Xero Technical discussion

In June 2013 we made a change to the way emails are sent from Xero. Some of our users have asked us why the change was necessary and whether we are planning to provide more options in the future. Be warned ­ this is a fairly technical topic so take a deep breath.

Xero allows organisations to raise invoices and statements and send them to their customers.

Sending on behalf of hundreds of thousands of Xero organisations, to the millions of their customers is a significant undertaking, and one that we take seriously.

Recently we changed the mechanism by which we send customer invoice emails. This post is an attempt to describe the new setup, the reasoning behind the decision, and a road­map for future changes that we hope to make.

Setting the scene:

The email landscape has changed dramatically over the past 5 years, and mail servers and email ISPs have all developed unique practices to try and shield their users from spam and other unsolicited email that currently floods the internet.

Over the past 10 years, several standards and de­facto practices have evolved to help mail servers make decisions on which emails to trust:

● (SPF) ­ a 10 year old standard that lets the owner of a mail domain (e.g. gmail.com) specify which mail servers are allowed to send email ● DomainKeys Identified Mail (DKIM) ­ a slightly newer standard that allows the sender of an email to digitally sign their email to prove that it came from them ● Grey listing ­ a heuristic that some receiving mail servers use to slow down the receipt of email, on the theory that spammers won’t attempt to queue and retry messages ● Spam filtering ­ a set of algorithms that receiving mails servers can run on email to try and determine if the content of the email is “spammy” ● Blacklisting ­ various databases of IP addresses or email addresses that are known to send spam ● Domain­based Message Authentication, Reporting & Conformance (DMARC) ­ a recent specification that combines SPF and DKIM to allow email senders to prove that the mail was sent by them

We realise that our users absolutely, positively want email that they send to be received by their customers ­­ their cashflow and business depend on timely payments. Even though email is by design an unreliable communication mechanism, we are trying to make it as robust and reliable as possible for Xero customers to use email to send invoices to their customers.

Our previous approach:

Our previous approach to sending invoice emails from within Xero was to “impersonate” or send email “on behalf” of the user currently logged in to Xero. e.g. If [email protected] had signed up to Xero, and proved that they controlled the , then we would happily send out email that looked like this:

From: Joe Bloggs To: A Customer Subject: Here is your invoice, pay me now!

In 2006, this was a fairly reliable solution, and most mail servers would happily accept these emails and deliver them to the recipient.

In the subsequent years, it was possible for Joe Bloggs to add an SPF record to their flatnotfound.com to indicate that Xero was able to send email on their behalf. This worked reasonably well for a few more years.

The Problem:

It is now no longer possible for SaaS vendors to reliably send email “on behalf” of their customers.

If a recipient mail server receives an email like the above example, the mail server will perform some checks to try and decide whether this email was legitimately sent by Joe Bloggs.

One of the indications the server will use is the IP address of the server that delivered the message to it. If the server does not look like a legitimate server for Joe Bloggs’ ISP (flatnotfound.com), then that’s a negative mark against the email. Traditionally a combination of reverse DNS lookups and SPF would be used to perform this check.

These days, setting SPF records is not sufficient to prove the authenticity of an email, and this example email illustrates the reason why:

Envelope Sender: [email protected] From: Joe Bloggs To: A Customer Subject: Here is your invoice, pay me now!

The “Envelope Sender” address is never displayed to regular email users, but it is actually the address that SPF uses to test the authenticity of an email. In this example, if this email message arrived from a server that was legitimately run by “evilspammer.net”, then technically that’s all that is needed to be legitimate from an SPF perspective ­­ even though the email displayed in the recipient’s mail client doesn’t indicate it comes from evilspammer.net!

For this reason, SPF is no longer the solution to sending email “on behalf” of customers like it was when we launched Xero a few years ago.

We have had this confirmed by email experts from Facebook, Yahoo! and JPMorgan Chase when we spoke to them in February this year. In fact, they confirmed what we were seeing, which is that there’s no way to reliably send email while pretending to be someone else.

The previous method of sending Xero emails out by impersonating the Xero user was untenable, and we would see continuing increases in the numbers of emails that would fail to deliver.

Problems with SPF:

We’ve already discussed how SPF on it’s own is not sufficient, and that it relies on the verification of a part of the email message that’s not actually displayed to an end user.

Our invoice emails actually looked roughly like this:

Envelope Sender: [email protected] From: Joe Bloggs To: A Customer Subject: Here is your invoice, pay me now!

For this email to pass an SPF test, the IP address of the mail server is compared to the SPF records of the “Envelope Sender” domain of notifications.xero.com. Xero has control of these records, and so we were able to make sure that the SPF record for notifications.xero.com was always valid.

Some mail servers would also run SPF­like tests against the visible “From” address that displays in the email, even though that’s not strictly part of the SPF standard, effectively asking the question: “Is Xero’s email server allowed to send email from flatnotfound.com?”

In order to pass this test, every Xero customer would need to create an SPF record in their DNS to allow Xero’s mail servers to send email on their behalf.

For some Xero customers that are IT­savvy, and in control of their own DNS, this was indeed possible, and over the past few years approximately one hundred Xero customers have managed to do this successfully. However, for most Xero customers this is infeasible:

● Many Xero customers use an “ISP” email address, rather than their own domain name (e.g. [email protected] rather than [email protected]). This means that they have no control over the configuration of their ISP. ● Customers that had their own domain name (e.g. MyCompany.com) but weren’t able to change DNS, had a large company with strict IT policies, or were part of a multi­national. These customers aren’t easily able to create SPF records. ● Customers that use SPF, but didn’t include Xero’s servers in their SPF records may be hard­failing all impersonated mail from other sources. Our analysis showed this was potentially the case for about 7% of customers. ● Corporate customer’s mail servers would often reject Xero invoice emails that they sent to themselves, even if SPF was correctly set up. E.g. a corporate mail server for MyCompany.com would say “I’ve recieved mail from xero.com’s mail server’s that is pretending to be from MyCompany.com. I know that I’m the only mail server for MyCompany.com, so this is obviously spam” and throw it away. ● Creating valid SPF records is actually hard. One only needs to look around the internet and on our own Community to see that people don’t understand the nuances of SPF, and a simple mistake in configuration can stop delivery of all your company’s email. This goes against our mantra of being the “World’s easiest accounting system”.

The (2013) solution:

We absolutely want to deliver email for our customers, to their customers. Asking our users to download invoices and send them from their desktop is cumbersome and inefficient ­ although we realise for some customers that it will always be their preferred approach so that they have complete control.

Candidate solutions:

● Continue impersonating email ○ Unreliable, not recommended by experts. ○ Requires every Xero user to configure SPF correctly, even though that won’t guarantee delivery. ● Send email via our customer’s email servers ○ Technically infeasible (we’d have to collect login credentials, connect to hundreds of thousands of authenticated mail servers to send email, and deal with the myriad of different ways that could fail). ○ We’d be outsourcing the “email delivery” issues to our customers, rather than being responsible for delivering mail correctly on their behalf. ● Deliver email with a .xero.com “From” address ○ We can guarantee that messages are correctly authenticated. ○ We control the full infrastructure for the email sending. ○ We can track deliverability and bounces.

Ultimately, we made the decision to choose the third option, and change the way that we deliver invoice email. We did consider a hybrid approach, or allowing users to opt­in to one or other approach, but we decided that there were benefits in having a single, reliable delivery mechanism that required no configuration by our customers.

The change we made in June 2013 changed the format of our invoice emails to look like this:

Envelope Sender: [email protected] From: Joe Bloggs Reply­To: [email protected] To: A Customer Subject: Here is your invoice, pay me now!

To break down the individual changes, and the rationale behind them:

The “Envelope Sender” address, remains as a unique @notifications.xero.com address. This is the domain that is used in SPF to validate that our mail server is allowed to send this email. This address is not displayed in user’s email clients, and the unique part at the beginning of the email address allows us to track bounces and failures back to the individual sender.

The “From” address has been broken into two parts: ● The Display Name “Joe Bloggs”, which is now configurable per organisation in Xero ● The email address message­[email protected], which is no longer configurable

The Reply­To address is now configurable per organisation (e.g. [email protected]). This is the address that any customer replies to your invoice emails will be directed to and displayed to them, if they press the Reply button in their mail client.

Our testing showed that in many mail clients, the recipient of this email would see nothing untoward, and simple actions like reading and replying to emails would not display the “xero.com” mail addresses.

In some mail clients (e.g. Outlook 2013), the new email format means that recipients can now see “xero.com” in the invoice emails that you send. This is an unavoidable side­effect of this technical decision, and it was not made for marketing reasons!

The reason that some mail clients display “xero.com”, is that those clients want to show the user the information they have used to make the “trust decision” that has lead to the email not being marked as spam. Our new email format leverages xero.com’s trust and email delivery mechanism, so Outlook wants to make it clear to the user who receives the email the reason why it has appeared in their inbox. I’ve labelled this as “The (2013) Solution”, because although this the best solution at the moment, the nature of the evolving war against spammers means that this approach is not guaranteed to work forever.

Bounce messages

There are several different types of replies that might happen to an email.

If a user is performing the reply, usually by clicking the “Reply” button in their mail client, then the reply will go to the “Reply­To” address, and arrive directly in the original sender’s inbox.

If a user is using an old mail client that doesn’t understand the “Reply­To” address, or they manually reply to the “From” address of message­[email protected], then their reply won’t be sent to the original sender, but will rather be sent to the unmonitored mailbox hosted by Xero.

We don’t expect that this will happen too often, but in those situations that it does happen we want the experience for your customers to be understandable. So what we’ve done is have a robot “Auto Reply” to all of those messages with an “Oops!” message asking them to re­send their email to you rather than Xero:

This mailbox is unmonitored, and we don’t read any of the emails that are erroneously sent there. Some time after the auto­reply is sent back to the original sender, the email will get deleted from our servers.

The other types of replies that will come from an invoice email are sent automatically by the recipient’s mail server, and called “Bounces”. Bounce messages will usually happen in one of two ways:

● An immediate failure, which we’ll send directly to the user (e.g. domain name doesn’t exist, or has been mis­typed) ● A slower failure, where the recipient mail server initially accepts the mail message but then later decides it won’t deliver (e.g. the username doesn’t exist, or their mailbox is full). We receive these bounce message to the Envelope Sender address of [email protected], and re­route it back to the original sender at the Reply­To address.

Implementing the DMARC standard:

Those of you that run your own mail servers will know how hard it is to track the deliverability of email, and to diagnose failures that have occurred.

Thankfully, the new DMARC standard allows mail senders such as Xero to receive delivery reports from the recipient mail servers such as GMail, Yahoo! or Hotmail, and actually diagnose any issues that might have occurred.

We are very excited to have partnered with Agari to begin the process of moving all our invoicing email to conform to the DMARC standard. Agari were a part of the original DMARC working group, and have worked with some enormous US brands to share their advice and implement cutting­edge email trust solutions.

The DMARC standard allows us to conclusively prove to recipient mail servers the invoice emails we send have come from a legitimate source. It blends the older SPF and DKIM standards with a new “alignment” process to make sure that their protection messages apply to all aspects of an email, not just the Envelope Sender address.

The other thing that DMARC and Agari allow us to do is to track trends in our delivered email, and to work on errors in our configuration as reported by the recipient. We’re now at the point where Google, Hotmail and other mail providers can send us daily reports telling us why they rejected email, and of any issues that they’ve had delivering email to their users. This is unprecedented and a completely new approach to give us visibility into what happens to the email we send once it leaves our server.

Our Roadmap:

Changing the way we send email is difficult, but also risky. We absolutely want to avoid making our customers’ invoice emails fail to deliver. To this end, we’re gradually phasing in our adoption of DMARC and the changes to our email sending process.

Step 1: Monitor email delivery using Agari tools May 2013

We have implemented a DMARC policy in report­only mode, which allows us to track the changes we’re making without instructing recipient mail servers to reject any email that doesn’t conform.

Step 2: Change the from address of outgoing invoice emails to a @post.xero.com address. June 2013

This lets us use a unique domain name for invoice emails that isn’t reliant on the “reputation” or delivery characteristics of any other email that Xero sends. It also allows us to introduce simpler SPF and DKIM policies, and DMARC policies, because any changes will be restricted just to email that comes from a post.xero.com address.

This also means that the visible “From” address now aligns with the “Envelope Sender” address, as they are both xero.com sub­domains. (Domain alignment is a DMARC requirement)

Next steps:

Step 3: Implement DKIM signing of post.xero.com email

This will allow us to cryptographically sign all invoice emails, so that a recipient mail server knows that they definitely came from a Xero server and aren’t spam.

Step 4: Turn on DMARC enforcement

This will allow us to instruct recipient mail servers to start enforcing our DMARC policies, meaning that they will definitively know whether email we’ve sent is from Xero, from a mail server we control, and hasn’t been tampered with. This will allow mail providers such as Gmail or Yahoo! to have a much higher level of confidence in our email, and will short­circuit a lot of their spam filtering.

The end result of these four steps will be that we’ll have a much higher quality of email that we’re sending, recipient mail servers will have a much higher confidence in the email they receive, and we’ll have industry­leading monitoring of the deliverability of the email we send, so we can be confident that Xero’s customers are sending invoices to their customers successfully!

Frequently Asked Questions:

Is it possible to have emails come from our mail domain rather than post.xero.com? Unfortunately, there is no longer a way to do this and reliably deliver email to your customers.

But it looks like Xero is sending the email to my customer, and it messes up their plugins and ruins our brand!

We understand, and sympathise with this concern ­ but currently don’t have a technical solution that will allow us to do this.

Is it possible to have all our outgoing emails appear in our Gmail Sent Items folder?

To do this, we either have to send out your invoice emails via your Gmail account (which we’re not able to do), or to carbon­copy (CC) your email account with a copy of every message you send from Xero.

We could consider making the CC of your email account on every Xero­generated email an option that you can configure, if that’s something that the community votes on as a good feature. In the meantime you can click the “Send me a copy” checkbox when sending invoices.

Another option we have is to track all outgoing messages in the Xero Notifications Inbox, so that you can view all messages sent from your account there.

Rapportive and other mail client plugins don’t display our company information any more

This is because these tools are looking at the displayed “From” address to determine which company has sent the email. Unfortunately there is no way we can change the way we deliver email to allow these plugins to treat email originating from the Xero mail servers as coming from you.

Is it possible to change the email address from message­[email protected] to [email protected] to make it more obvious it’s unmonitored?

That’s an interesting question, and one we considered when setting up the From address. We may consider changing this in the future.

Does Xero track the deliverability of our email?

Yes. We are in the early stages of using the Agari platform to provide us with metrics and dashboards showing us the deliverability of the invoice emails you send using Xero.

Please note that this is not a privacy­invading tracking system like the hidden image / web bug approach used by some delivery tracking systems, and is purely a report delivered from the recipient mail server (e.g. GMail) to our secure Agari environment. Does Xero return bounce messages to my email address?

As mentioned in the Bounce section above, there’s two different types of bounce messages that we may receive in response to your invoice emails, as well as other replies that might come to our servers rather than directly to you.

For all of these message types we’ll attempt to redirect the bounce or reply back to your inbox. In those situations where that’s not possible the other email address will receive the “Oops!” message above. However note that there are many different ways for email to fail silently and, in some cases, neither us nor you will be notified by the recipient mail server.

Why haven’t you set up SPF for post.xero.com? Is the SPF record for xero.com valid?

The domain used for SPF checks of our invoice emails sent through Xero is notifications.xero.com. The SPF record for this domain is relatively simple, and we have had it’s configuration checked by external parties.

Are more messages being treated as spam than before these changes?

With any change to the way we send email, we’re very cautious and want to make sure that we’re making things better rather than worse.

Unfortunately, before embarking on this project with Agari and the DMARC standard, we weren’t able to scientifically track deliverability of email like we are now able to. We were however able to unscientifically gather metrics about email delivery issues through support tickets raised by our customers, and we’re confident that these have decreased since the changes.

Going forward, we look forward to being able to collect better metrics on email delivery, and depending on the community demand we may even be able to share some of these metrics with Xero customers.

Could we give each Xero customer their own sub­domain (e.g. my­company.xero.com)?

That’s an interesting idea, and not one that we’ve yet explored in depth. In that case, we’d need to provide email signing, DKIM, SPF and redirection for hundreds of thousands of custom domain names, potentially one per customer. We’re unlikely to do this in the short term.

I previously used another desktop product, and it was able to send email on my behalf. Why can’t Xero do this?

Most previous generation or desktop products that sent email would typically send the email directly from the logged­in user’s computer, using their local Outlook or Mail preferences. Because Xero is a Software­as­a­Service application running on our servers, this model is not possible ­ our servers need to be able to send your invoice email even if your computer is not connected to the internet.

You could potentially revert back to sending your invoice emails from your desktop by downloading the PDF files and sending them from your local Outlook or Mail client, or send the invoice emails to yourself from Xero and forward the invoice emails to your customer manually from your mail client. However, this process will be more time consuming.

I use another SaaS product, and it is able to send email on my behalf. Why can’t Xero?

Our review of other SaaS applications, and the ways they send email on behalf of the users (impersonate), shows two main approaches:

1. Incorrect emailing practices, where impersonated email will fall foul of the SPF or spam rules we’ve discussed above, much as our 2006­2013 solutions did. 2. Correct emailing practices, but which require the customer to set up SPF and DKIM records in their DNS, and don’t allow email impersonation until the SaaS provider has confirmed these settings are correct.

We now clearly hear that some of our customers would like the second option, and to be able to opt­in to the extra configuration work to configure their SPF and DKIM to allow us to send on their behalf.

We are currently evaluating the provision of this feature, and the engineering and support effort required to provide it. We have run some analysis on the number of customers we expect would be able to do the required configuration to make this work correctly, but it would be still great to hear your feedback via our Xero Community site.