Never forward email to GMail. Ever. Google ’causes’ backscatter.
What is Backscatter
These days, one of the main ways spammers get into your mailbox is through a technique called ‘backscatter’. Backscatter happens when a mail server accepts an email and later decides that it can’t deliver it, and so informs the sender through a bounce. Unfortunately, what spammers do is they send their spam to accounts they know do this with a bogus Reply-To field (that’d be you). Thus you get, in your inbox, bounces to email you never sent, usually containing some kind of Cialis or Windows XP For Free scam. In theory, the spammers are using the relatively high trust level of a legitimate mail server in order to get spam to you, where an open relay (the previous method of choice) would get them rejected quite quickly.
Nowadays, though, allowing your server to send backscatter is a one way ticket to no-trust zone. You’ll get added to all the lists if your server is believed to do this. So, in the end, the spammers are back to square one — but they don’t really care.
The solution is relatively simple — for the most part. If you can’t accept and deliver an email, your server should reject it immediately. All of the major mail servers are now capable of doing this out of the box (except, unfortunately, qmail which requires plugins to do this correctly. But no one runs a vanilla qmail install out of the box anymore).
GMail
Unfortunately, there’s one situation where this becomes a difficult again. This situation is when you’re forwarding. When an email is forwarded, say from my host at stormbrew.ca to stormbrew@gmail.com (for illustration only, this is not my configuration), stormbrew.ca believes it can accept this email. It knows that it will then need to forward the email, but it has no reason to believe it can’t do this.
So it accepts it, adds it to its queue, eventually decides to send it to gmail, and then waits for gmail to give a response.
And this is where it becomes impossible to prevent backscatter. Normally, the reasons for rejecting an email would be largely permanent (mailbox doesn’t exist) and thus fixable by simply changing the forward to an account that does work, or temporary (server is overwhelmed and wants you to back off on sending for a while) and can simply be solved by your mail server deferring its email.
Where gmail makes this an exceptionally difficult problem is in how it rejects mail that it considers to be spam. Most mail servers (mine included) silently reject spam. Whether at the point of reception, or later after the mail has been queued, but without an error or a bounce. GMail, on the other hand, will loudly reject the email immediately as an error:
2011-02-15 02:46:42.363718500 delivery 23672: failure: ##.###.###.##_failed_after_I_sent_the_message./Remote_host_said:_550-5.7.1_[##.##.###.###_#]_Our_system_has_detected_that_this_message_is_likely/550-5.7.1_unsolicited_mail._To_reduce_the_amount_of_spam_sent_to_Gmail,_this/550-5.7.1_message_has_been_blocked._Please_visit___________________________/550-5.7.1_http://mail.google.com/support/bin/answer.py?hl=en&answer=188131_for/550_5.7.1_more_information._r13si6986867yhh.42/
It will, however, continue to accept email that does not trigger its spam filter. So this behaviour is neither temporary nor permanent in the traditional sense, though the error message indicates a permanent error.
Upon receiving this error code, my mail server will send a bounce — the absolutely correct behaviour since in normal circumstances you’d want to know if the forwarded mailbox was full or no longer existed. But this just plays into the hands of the spammer, giving them another free port to send spam to/from.
Please note that this is true regardless of whether or not you follow the advice at Google’s page on forwarding best practices. That only affects whether your host will be identified as a spam-source directly due to this behaviour, and even if you filter spam before sending it on to gmail it’s unlikely that you’ll be 100% successful by google’s standards (where they have much more computing power to throw at this problem). You will still likely wind up backscattering due to this.
Google should, if it does not want to accept these messages, silently bury them as pretty much every other mail exchange does. Spammers keep finding new ways to exploit attempts to reply to spam to indicate its spammyness, and this is no exception.
Solutions
As the title suggests, you should try to avoid forwarding your email to a gmail account if at all possible. If you can, and you intend to use gmail as your email UI, you should probably use Google Apps to run your domain through.
Another, less extreme, solution if you only want it for an account or two, is to use gmail’s ability to pull from your server rather than pushing to it. I have never set this up, but I know others have done it and I don’t expect it’s too difficult. It does require that your mailbox be exposed as POP3, as far as I know.
I’d really like to see Google drop this method of rejecting spam, though. I see what they’re trying to do, but I strongly feel that it makes them a poor email citizen (though they are a great one in many other ways).
February 17th, 2011 at 1:23 pm
Google is doing exactly what it should do, and what is standard industry practice… Silently dropping mail is generally considered to be bad, and I don’t know any major email service providers that do it. Hotmail *sometimes* does it, but usually doesn’t.
Rejecting with a 5xx code is the correct response when you don’t want to accept a message. It is your server that is generating the backscatter by accepting the message and then ultimately bouncing it.
Email forwarding is just generally not very good. If you can avoid it, do so. Perhaps you could use Google Apps for your Domain?
February 17th, 2011 at 1:33 pm
I disagree that rejecting email loudly for being spam is ‘standard industry practice.’ The vast majority of spam filters happen at a layer deeper than the SMTP accept point where they can be rejected immediately. Rejecting mail immediately for not being to a valid account or other purely administrative reasons is absolutely industry standard practice, but this is different.
Also, in the case of spam, silently dropping mail is considered *good practice*. If you discover mail is spam (very common) after it has been accepted by SMTP and do *not* silently drop it, that means sending a bounce. Sending a bounce is how backscatter got started to begin with.
My mail server is a multi-user mail server, and as such I don’t have direct control over how my users want to use their email. I definitely intend to advise them to use either POP-pull or google apps (as, you might note, I specifically said in the blog post at the end), and if they don’t I will have to disable their forwarding. Hence the title, “Never forward to gmail.”
February 17th, 2011 at 1:50 pm
Strange. I’ve been running large mail systems for ISPs and Universities for years and have been on various high volume email administrator mailing lists with hundreds of other mail admins, and what I stated in my original comment is what I recognise to be standard industry practice.
I’ve never seen anybody recommend silently dropping mail as a good thing and have seen it described as bad behaviour countless times. Spam filtering at the edge, during SMTP and providing 5xx rejection errors is the correct behaviour. If you just silently drop mail, neither the sender, nor the recipient knows that there was a problem.
February 17th, 2011 at 2:16 pm
“Silently drop” is probably too strong a term. Though I would argue that “dump into a spam folder and delete it after 10 days” is close enough.
Again, though, I’m not aware of any major email provider that has as a policy to reject at the SMTP level spam *other than* gmail. I have just tried with both hotmail and yahoo, sending a message who’s subject and body consisted mostly of variations of viagra, cialis, and vacation, along with a few $s. Both hotmail and yahoo immediately accepted the mail and put it in the spam folder. This does not bode well for considering smtp rejection of spam as ‘standard industry practice’.
I’d be curious to know what servers you know of or administer that do something similar to gmail. I would still argue that any reply indicating that a message was rejected *for being spam* plays into the hands of spammers, as I can’t figure any sane policy to treat *this* error as one that should not bounce while treating ‘real’ errors (mailbox full, user does not exist) as ones that should be bounced. If you can come up with a reasonable policy to achieve that goal, I’d be happy to hear it.
February 17th, 2011 at 2:39 pm
Exim is one of the largest MTAs on the Internet. It’s standard when using Exim to perform spam/virus scanning in the ACLs during the SMTP stage and reject there. SpamAssassin and ClamAV and other scanners are called from the Exim ACLs during the SMTP phase.
AOL’s spam filter rejects with a 5xx during SMTP. Pipex and Webfusion used to and I assume still do. Most UK Universities seem to. I could come up with thousands of examples.
February 17th, 2011 at 3:12 pm
SMTP time rejection is industry best practice. You trigger the rejection threshold, you don’t get in. If you look a bit dodgy but *don’t* hit the threshold, you go in the spam folder.
Postini, MXlogic, Barracuda, Roaring Penguin, Symantec, SpamAssassin with Sendmail/Exim/Postfix etc etc, Rogers, Sympatico, Worldcom, Yahoo!… the list of software or networks which reject is almost endless. Reject, don’t accept, because that makes the sending MTA have the liability of generating the NDR – and this is your problem.
You’re accepting stuff and forwarding it; if the recipient rejects it then that’s your problem because you already determined in some way that the message was “good” (for some value of “good”) and accepted it.
There’s another way to look at this: your network, your rules. Which translates to my network, my rules; their network, their rules. If they reject, that’s because they think the message is spam, and you have to deal with that,
Trust me (well, you don’t have to): send something to Yahoo or Hotmail/Live enough times to get in the spam can, and you’ll find yourself being deferred. Do it $enough_times++, and you’ll get blacklisted temporarily. The problem is that from outside, you’ll never know where the threshold is.
February 17th, 2011 at 3:47 pm
You must never drop mail silently.
Obviously you shouldn’t accept the mail and later bounce it either (note: bounces do *not* go to the address in the Reply-To: header, as you strangely seem to think; they go to the reverse-path given in SMTP).
You should reject at SMTP time instead, just as Gmail and everyone else does. That is not only ‘industry best practice’ as Graeme points out, it is the *only* acceptable practice.
If you drop silently, then *genuine* senders don’t get informed when their message doesn’t get through, whether it’s because they typoed the recipient address or their mail suffered a false positive in your spam filtering. By dropping mail silently you are making the system unreliable — effectively throwing the baby out with the bathwater.
If you want to forward mail, it’s up to you to ensure that you don’t accept messages which the recipient is going to reject as spam. Make sure your own spam filtering is at least as good as the filtering of the final destination.
February 17th, 2011 at 10:02 pm
Thanks for the input, guys. I remain unconvinced that this is a *good* practice, but when the whole world tells you it is *standard* practice you have to listen. I’m a programmer by trade and a sysadmin by hobby, so I won’t argue the point too strongly. My mistake can be forgiven, I hope, by the fact that gmail is the only one that I’ve seen this behaviour as an observable one so far. And I suspect they’ve raised the threshold one of you was referring to in recent times, as (for me at least) the ability to even see a spam folder on gmail seems to have disappeared.
Whether my reasoning is correct, though, would you agree that forwarding email to gmail, as opposed to using the alternatives outlined in the last paragraph of my post, is probably not a great idea? Especially given that google gives you those options?
Again, thanks for the input. I’ll probably write a followup post in the next few days.
February 18th, 2011 at 2:21 am
GMail pulling from your server via POP3 is a much better solution yes. Email forwarding on the Internet has never particularly worked well.
Another common problem that it causes is getting your server blacklisted for forwarding spam. As far as the destination server is concerned, it’s your server that is sending the spam.
You also have the problem with SPF. If the destination server does SPF checking, and the original sender has strict SPF records on their domain, then SPF will fail. As the recipient mail server will see an email coming from a host which it’s not allowed to. The solution for this one is rewriting the sender envelope to an address on your own domain when forwarding on.
February 19th, 2011 at 9:29 am
One comment that was made was “your network, your rules.” There is a scope consideration that has been alluded to but not stated clearly. If you run your own personal mail server with your own personal domain and you are the only consumer of email, silently dropping is effective (for you). When you provide paid services for others, you cannot make quite so much of a sweeping decision. Undoubtedly some of our 20K users actually did sign up for mailings from $VENDOR that *I* would consider spam, but silently dropping their wanted emails is a quick way to end up with substantially less than 20K users and “I encourage all my competitors to do so.” Sign up for and use feedback loops to tune your filters because one of your customer forwarders to an AOL box that’s later submitted as spam is still feedback from YOUR customer, just through someone else’s system. Use sender reputation as part of the decision. Use RBL’s. Use selective greylisting (for non-reverse resolving IP’s, for hostnames that match dynamic patterns, etc). Use everything available to you that allows you to make a better decision of ok or spam at smtp time.
A daily occurrence at $JOB is “customer didn’t receive an email” and I have to go find it in the logs where hotmail or gmail or $ISP accepted the email in question. Then it’s the customer’s issue with the ISP, not us.