email forwarding considered harmful

Date: 2007-06-19 21:16:00

Email forwarding is a problem no matter which way you handle it. The problem arises from the increasingly widespread use of SPF. SPF is supposed to be optional, but an email forwarder that doesn't do anything about SPF can still run afoul of it.

I have set up a few addresses that forward from domains I own, to addresses at gmail.com or yahoo.com or wherever. I have set them up using Postfix virtual domains. My Postfix server receives and accepts the email, then turns around and resends it along to (for the sake of example) yahoo.com. The message headers and SMTP envelope remain unchanged except for the addition of a "Received:" header.

When yahoo.com receives the message, it will do an SPF lookup on the SMTP envelope-from domain to find out whether the connecting SMTP server (my server) is allowed by the sender's SPF policy. If the sender does not publish SPF, then the test would be assumed to pass. However, if the sender publishes an SPF policy, it's not likely to include my server, and therefore yahoo.com may recognise an SPF fail situation.

For the sake of simplicity, consider two messages: A legitimate one from somebody at sender.com, and a spam message with a forged header from somebody at forged.com.

Simple mail forwarding
Situation	Result
Legitimate message from sender.com (without SPF)	Message accepted with no adverse effects.
Legitimate message from sender.com (with SPF)	Message fails SPF test, possibly rejected.
Spam message apparently from forged.com (without SPF)	Message accepted, but yahoo.com filters the message as spam and my server is marked as a spam source.
Spam message apparently from forged.com (with SPF)	Message fails SPF test, my server is marked as a spam source.

This state of affairs doesn't look so good for me, the guy stuck in the middle. This has been a long standing problem, and Sender Rewriting Scheme (SRS) is supposed to be a way to fix it. Under SRS, my forwarding server would change the SMTP envelope-from to a (specially constructed) address under my own domain. In general, this would ultimately cause yahoo.com to not process an SPF fail on messages forwarded from my server, since the SPF test would be run using my own domain's SPF record. However:

Mail forwarding using SRS
Situation	Result
Legitimate message from sender.com	Message accepted with no adverse effects.
Spam message apparently from forged.com	Message accepted, but my server is marked as a spam source. Also, since my server has accepted some responsibility for the message by using SRS, my domain may be tagged as a spam source too. There's nothing in SRS that says spammers can't forge it too, as far as the receiver is concerned.

Using SRS makes the situation where a legitimate sender.com publishes an SPF record, not so bad for me as the forwarder. This of course is the situation for which SRS was designed. However, in the case of a forged-sender spam message, it's a lose-lose situation. SRS does not help in this case. The only thing I can do is only forward legitimate mail, and block all spam mail. If I could do that automatically and reliably ... well, I can't.

The above situation is not hypothetical. My mail server currently has trouble sending to yahoo.com (messages can be delayed for hours), because there has apparently been enough spam relayed through my server in the direction of yahoo.com for them to mark my server as a spam source. The only thing I can do is try to implement stronger automatic spam-blocking on forwarded email. If you run a mail forwarder and you forward spam, you're screwed.

Comment

edm : Mail forwarding
2007-06-19T11:01:54Z

I generally use procmail to forward mail between accounts (rather than virtual domains or similar) in part because it rewrites the envelope header (and thus avoids the SPF issue) and in part because it lets me choose to, eg, only forward things which haven't been marked as suspected spam.

Unfortunately that doesn't completely solve the problem. Because as you've noticed if you forward enough things that someone else detects as spam then you too will be marked as a spam source (or spam relay), and it's all downhill from there. (And my spam detection is very definitely less than 100% effective -- I'd say it's probably less than 75% effective at this point.)

I don't think there's an actual solution to that problem other than, well, don't forward mail. Which appears to be where we're heading as a long term position. And I've come to this conclusion very reluctantly, as email seems to be at its least useful in the last 20 years. No wonder IM is at an all time high usage in work contexts.

Ewen

cetan : Re: Mail forwarding
2007-06-19T13:05:38Z

[[No wonder IM is at an all time high usage in work contexts]]

I wish someone would tell my company that so they'd stop treating IM like some sort of WMD...

ghewgill : Re: Mail forwarding
2007-06-19T21:29:18Z

I'm wondering whether using Spamassassin in a milter situation with Postfix (with a higher threshold) might be able to effectively eliminate even more spam before being forwarded or otherwise handled. My SMTP stats indicate that even after all the Postfix checks for SMTP protocol violations plus SPF checks, Spamassassin still is responsible for blocking over half the incoming spam to my mailbox. I already separate spam with score >= 10 from spam with score >= 5, and have never detected a false positive in the >= 10 folder.

It is certainly disappointing to see that email might be edging toward a spam-filled wasteland like Usenet before it.

edm : Re: Mail forwarding
2007-06-20T00:22:53Z

I've been using Spamassassin (and ClamAV) in a milter config with Postfix on some of the newer mail servers I've set up, and it seems to work fairly well. Certainly better than the Postfix/Amavis-New/Postfix pre-queue filtering setups on the older mail servers. I've been throwing away spam that scores over about 10 for a long time (and viruses too) without problems. (The problems with mis-filtering I've had have always been relatively low scoring, close to the 5 threshold, or where I manually thought it was spam that had come through to my main mailbox due to unrecognisable sender and/or poor subject line. I gave up a while ago and started whitelisting addresses in my mail client -- everything else goes into a "dubious" folder, which is mostly spam.)

Ironically I don't think that Usenet is a spam filled wasteland. It's just an abandoned wasteland. Everyone left it for journals/blogs/RSS feeds and the like. The spam didn't help, but spam on Usenet never reached the levels of spam on email. (AFAICT well over 50% of all email delivery attempts are junk, probably over 75%.)

Ewen

taral
2007-06-19T15:51:09Z

I either use procmail to forward (rewriting envelope headers) or I use XCLIENT or similar.

taral
2007-06-19T15:51:50Z

The real solution is to extend SMTP to allow me to say "I don't want the message, send it to X instead."

ghewgill
2007-06-19T21:00:12Z

Indeed, that would be better in most situations. There is in fact an SMTP response code 551 that says "I don't handle mail for this user, try instead". However, section 3.4 of RFC 2821 indicates that the originating server has no obligation to recognise, handle, or even inform the sending user that the mailbox has a forwarding address.

But there are disclosure concerns with this approach - perhaps somebody would like to use forwarding to hide their actual mailbox address. Also, a 551 code couldn't handle forwarding a copy of the message.

taral
2007-06-19T21:05:18Z

In the case of forwarding for privacy or obscuration, one can reasonably assume that the message is being accepted by the forwarder as proxy for the real email owner. Therefore any spam checks must be performed by the forwarder, and then the envelopes and headers rewritten to indicate that a trusted sender (the forwarder) is now sending the message.

ghewgill
2007-06-19T21:18:48Z

Therein lies the problem, of course. One must be highly confident of the antispam solution in use, because false negatives leak spam through to the final destination, and false positives may be lost completely. It would be possible to set up an intermediate local spam mailbox so people could sift through the junk and look for false positives, but that would be a lot of work.

taral
2007-06-20T00:38:31Z

My point was that relays should rewrite messages to mark them as "trusted" so that they are not subject to spam filtering at the new recipient. This fixes most of the problems you listed in your entry.

cowbert
2007-06-20T07:07:34Z

what about domainkeys?

taral
2007-06-20T23:11:40Z

I'm not familiar enough with it to comment.

ghewgill
2007-06-20T21:08:50Z

But how can I do that in such a way that the result will be acceptable to yahoo.com? I don't have any control over how yahoo filters its mail. I can make SPF pass, I can DKIM sign the message, I can do any number of things to prove that the message came from me, but I can't do anything to change yahoo's view of whether the message is spam or not. And if it is spam, then all my assurances that it came from me just reinforces yahoo's view that my mail server is sometimes a spam source.

taral
2007-06-20T23:11:19Z

Well, I don't know for certain, but many of the services will not mark things as spam if they are "whitelisted" by being added to your address book.

ghewgill
2007-06-21T00:15:05Z

That's all well and good, but the final recipient whitelisting a sender address doesn't involve me as the intermediate forwarder. The effect I'm seeing from yahoo is prolonged rejection of connection attempts by my server, long before my mail server even has an opportunity to send the message and identify the sender or recipient. The more spam I automatically forward, the worse this is going to get.

taral
2007-06-21T01:13:35Z

Sounds like a problem with using yahoo to me.

taral
2007-06-21T01:15:49Z

Also, what you say seems to imply that you're forwarding to other people, and they aren't adding your forwarding envelope address to their address book. Is that right?

ghewgill
2007-06-21T01:46:23Z

Yes, I'm forwarding for other people. On forwarding mail, my mail server isn't doing any rewriting of message or envelope headers, so there isn't any way for the yahoo user to ask yahoo to "approve" forwarded mail from me. My address appears in neither the message headers nor the envelope, only as the sending SMTP server.

That's not really the problem though. The problem is I'm forwarding all incoming email for a number of users. Some of this email is spammy. Yahoo notices that this spammy email is delivered from my IP address, no matter what header rewriting I might do to try to appease it. Therefore, yahoo starts treating my IP address as a potential spam source. If I try to take the next step and digitally sign the forwarded mail using DKIM or something, then yahoo knows the spammy email has come from me because I've explicitly taken some responsibility for it. The ultimate solution would be for me to have a perfect spam filter, and as we all know that's the Holy Grail of email.

taral
2007-06-21T02:14:17Z

No, the ultimate solution is for Yahoo to allow the recipient to mark your server/domain as a "trusted forwarder" and therefore not penalize you for sending the message.

cowbert
2007-06-19T16:34:28Z

And it's a giant pain in the ass to get yahoo.com to de-graylist you.

ghewgill
2007-06-19T21:05:40Z

Yeah, I'm seeing that. I filed a report as indicated in the 451 response code I was getting, but even after their response have seen no appreciable difference in behaviour.

decibel45
2007-06-22T00:33:12Z

Hmm... my server has appeared on a few RBLs, and I couldn't figure out why. But for a while I was forwarding all my email to google... I bet they though I was a spammer and reported me.

It might be better if you run stuff through SA before sending... hopefully that would cut out enough spam so that they didn't think you were a spammer anymore.

Greg Hewgill <greg@hewgill.com>