?

Log in

No account? Create an account
SPF and why some people are having trouble sending me email. - AdrianG [APOD]
January 8th, 2005
09:24 pm

[Link]

Previous Entry Share Next Entry
SPF and why some people are having trouble sending me email.
I'm not certain that every case where someone has had trouble sending me email is a case of SPF related problems, but the two cases for which I've been able to collect any data seem SPF related.

SPF stands for "Sender Policy Framework". It's an anti-spam measure, and I have mixed feeling about it. Every anti-spam measure I'm aware of involves a risk of misidentifying legitimate email as spam. The ISP that handles my email put this anti-spam measure into place without asking me, but in their defense, SPF generally has to be employed at the SMTP server that initially receives email for a domain. As such, it is probably not practical to try to employ this measure for some customers and not for others, and this ISP does have many other customers. In addition, since this measure was employed, my daily ration of spam has dropped from more than 150 useless, offensive, unwanted messages per day from Low Life Spammers to fewer than 10.

Having made excuses for SPF, I have to confess that my own configuration for sending email is not compliant with that standard. There are cases where email I've sent has bounced because SPF records for my domain don't indicate that my IP address is authorized to send email from "nerds.org". I have some advice for anyone who has had email to me or to anyone else bounce because of SPF problems. And I have a feeling that SPF record checking will become more and more common, so I imagine I will have to follow my own advice before too long. 8-)

First, for those trying to send things to me, the ISP that handles my "nerds.org" account is separate from the ISP that handles my broadband connection. The broadband ISP, Time Warner, supports some email addresses for me that I have never used, before Yesterday. One of them is "adriangr1" at "kc.rr.com". You should, of course, join those two strings together, without the quotes, and with an at sign in place of the word "at". Time Warner has their own anti-spam measures, and it's possible that you might stumbler over them, but they do not appear to use SPF.

Second, if you decide to solve the SPF issue, once and for all, understanding a bit SPF would help. SPF is designed to help fight the Unholy Alliance between Low Life Scum Spammers and Evil Virus/Worm Writers. In particular, this Unholy Alliance involves Evil Virus/Worm Writers infecting large number of systems owned by ordinary users and installing programs on them that allow armies of these infected systems to be used by Low Life Scum Spammers to send their spam. I'm simplifying things a bit, but the real point is, that spammers have been frustrated by efforts to blacklist their IP addresses to block their spam, and they've countered by taking control of large number of systems, without permission of the owners, and using those systems to send spam, thereby gaining a continuous supply of many new IP addresses from which to send spam, and making it impractical for anti-spam blacklists to keep up with all the sources of spam.

SPF tries to solve this problem by giving domain owners control over what IP addresses are to be allowed to send email in their name. As an example, "nerds.org" has a way of publishing the fact that email from any "nerds.org" address is only supposed to come from one of two IP specific addresses. In my case, I have my home computer configured to send email directly to it's destination, even though my home IP address is not one of the two authorized to send email claiming to be from "nerds.org". I'm not sending spam, but anyone doing SPF checking will reject email from me. The right solution is for me to reconfigure my system to send email through the ISP that handles my email, but it will require some research to figure out how to do this, and I just haven't done it, yet. As more and more places start to reject my email, I'll feel more pressure to take care of it. I'm guessing that most people don't have this specific problem, because they already use their ISPs to relay outbound email.

The other SPF related problem is that not every sending domain has SPF configured so that every receiving domain will accept its email. On the sending side, SPF lets a domain announce that email for the domain can come from only one IP address, from a range of IP addresses, from any combination of the two, or from all IP addresses on the internet. On the receiving side, SPF lets a domain specify how restrictive the sender's SPF configuration has to be before that receiving domain will accept the sender's email. In the early days of SPF, when it was still largely experimental, those domains that checked SPF records at all often followed SPF restrictions when the sending domain had them but accepted email from any IP address when the sending domain did not have SPF records. Now more and more domains are demanding that some sort of sender SPF configuration must be in place, and some are rejecting configurations that say "any IP address is okay." So, even if you are using your ISP to relay mail, if your ISP's SPF configuration is not restrictive enough, a growing number of receiving domains won't accept your email. It may be that all you can do, in those cases, is try to get your ISP to tighten up its SPF configuration, use another ISP, or see if there's not some way to get the receiving site to whitelist your address. Whitelisting means putting an IP address or email address on a list of addresses that get to bypass SPF checking. In the case of my ISP, it looks like the bounce message they send has an email address to which you can send mail to have your email address whitelisted. This sounds like it would make things to easy for the spammers, but the spammers generally doen't even try to process bounce messages, so it turns out to be a fairly useful approach.

Finally, I want to say that it looks very much like SPF is going to be more and more prevalent. Those of us (including me) who are not yet SPF compliant are going to have more and more trouble sending email until we have not choice to conform. Resistance will ultimately be useless or futile, depending on whether you are more of a Hitchhiker's Guide to the Galaxy fan or a Star Trek fan. 8-)

Adrian

(21 comments | Leave a comment)

Comments
 
[User Picture]
From:ms_tek
Date:January 9th, 2005 04:23 pm (UTC)
(Link)
Did you get my other email?
[User Picture]
From:adriang
Date:January 9th, 2005 04:49 pm (UTC)
(Link)
I got one email from you. It was a forwarded copy of the original one that bounced, I think. I replied to it, and as far as I know, that reply should have reached you. I'm hoping to look some stuff up and make a more complete reply, Today.

Adrian
[User Picture]
From:ms_tek
Date:January 9th, 2005 04:51 pm (UTC)
(Link)
=/ I didn't get it.
[User Picture]
From:adriang
Date:January 9th, 2005 05:07 pm (UTC)
(Link)
I tried replying again, and I found a message in my log saying it had been delivered. I've obscured some information in the message, and included it, here:
    Jan 9 10:49:18 cyclone sendmail[20999]: j09GnBhM020997: to=<xxxxxxxxx_xxxx@xxxxxx.net>, delay=00:00:03, xdelay=00:00:02, mailer=esmtp, pri=153387, relay=mx0.xxxxxx.net. [2xx.165.64.100], dsn=2.0.0, stat=Sent ({mx003} Message accepted)

It looks like this your MTA accepted this latest message. Did you get it, this time?

Adrian

[User Picture]
From:ms_tek
Date:January 9th, 2005 05:44 pm (UTC)
(Link)
ACK

make that

ms dot tekkie [at] g mail dot com
[User Picture]
From:adriang
Date:January 9th, 2005 06:03 pm (UTC)
(Link)
I've tried sending to that address.

Adrian
[User Picture]
From:adriang
Date:January 9th, 2005 06:03 pm (UTC)
(Link)
I should say, I've just tried.

Adrian
From:technoshaman
Date:January 9th, 2005 04:32 pm (UTC)
(Link)
So you're blocking anyone who doesn't have the requisite TXT records, done right? or are you taking the default interim, i.e. as long as the DNS is scrupulously correct on the connecting host, it's OK?

I killed off my SPF records and shut down my policy daemon after Microsoft got their fingers in the cookie jar. I expect I'll put something back once everybody gets their finger out and decides on something that's OSSG-compatible. (I'm NOT doing a proprietary solution; then again, I doubt AOL will let Microsoft do that.)

Meanwhile, I'm simply using almost but not quite the full range of sanity checking Postifx allows: I'm using two RBL's, making sure the connecting client is saying HELO with a real FQDN that is not my own, bouncing known quantity addresses no longer in use, and some of my own blacklists (and whitelists). I'm bouncing anywhere from 5 to 25 a day (it used to be a lot more, occasionally upwards of 100, but I think AOL and Comcast's little partnership have done a lot for that), and maybe two a week slip thru to be ground by bogofilter. Pretty darn fine, for me.
[User Picture]
From:adriang
Date:January 9th, 2005 04:57 pm (UTC)
(Link)
If I understand what my ISP is doing, correctly, they are blocking email from domains that don't have an acceptable SPF configuration, and they are rejecting SPF configurations that say all IP addresses on the internet are okay. I'm drawing this conclusion based on very little data, so it could be a little off.

I thought MicroSoft tried to do the embrace and extend thing but got rejected by the IETF. As far as I know MS has not been allowed to take control of SPF.

Just as an experiment, and you see if you can send me email?

Adrian
From:technoshaman
Date:January 9th, 2005 05:22 pm (UTC)
(Link)
Seems to have gone thru, at least stage one:

Jan 9 09:20:11 localhost postfix/smtp[28128]: 7DEDF4ADB1: to=
[Error: Irreparable invalid markup ('<adrian [...] nerds.org>') in entry. Owner must fix manually. Raw contents below.]

Seems to have gone thru, at least stage one:

Jan 9 09:20:11 localhost postfix/smtp[28128]: 7DEDF4ADB1: to=<adrian at-sign nerds.org>
, relay=ob.fu.sca.tion[192.168.0.0], delay=3, status=sent (250 message sen
t ok)
[User Picture]
From:adriang
Date:January 9th, 2005 05:27 pm (UTC)
(Link)
I get a message from you, but it looks like my reply won't make it.

Adrian
[User Picture]
From:adriang
Date:January 9th, 2005 05:28 pm (UTC)
(Link)
That should be "got". 8-)

Adrian
From:technoshaman
Date:January 9th, 2005 05:31 pm (UTC)
(Link)
Weird. Log from your end? I'll look at mine...
[User Picture]
From:adriang
Date:January 9th, 2005 05:40 pm (UTC)
(Link)
Jan 9 11:14:18 cyclone sendmail[21060]: j09HEGhM021058: to=<xxxxxxxxxxxx@xxxxxx.org>, delay=00:00:02, xdelay=00:00:02, mailer=esmtp, pri=120550, relay=mail.xxxxxx.org. [1xx.136.111.31], dsn=5.0.0, stat=Service unavailable

One of the consequences of my non-SPF compliant configuration at home is that I won't receive the bounce message.

Adrian
From:technoshaman
Date:January 9th, 2005 05:40 pm (UTC)
(Link)
Oh. You're doing direct-to-MX off a RoadRunner host. Does nerds.org not allow you to send via them?

In the old days I would've been all for everybody running their own mailserver wherever, but these days with the spam viruses running around the cable networks like fleas in a pound...

Actually, with my current volume of mail, and the way the stats have been looking lately, I could probably get away with removing that restriction... but still.

I really like the idea of having all my critical services hosted offsite, on a real T1, with backup power...
[User Picture]
From:adriang
Date:January 9th, 2005 05:53 pm (UTC)
(Link)
Actually, I'm sending directly from a sendmail instance here at home. That means I'm not in anyone's SPF records.

What I need to do is configure my local sendmail instance to relay through nerds.org's MX, and to use a password. The trouble is, if I configure it to do that, directly, then the next time I use YAST2, it will rewrite at least part of my sendmail configuration to what it thinks is correct. I've got to either figure out how to tell YAST2 what I want, or I have to modify the configuration build process for YAST2 so that it builds what I want. That's really what's kept me from fixing my SPF configuration so far. I'll get around to it, eventually.

Adrian
From:technoshaman
Date:January 9th, 2005 05:59 pm (UTC)
(Link)
*nods* Actually, if you don't have too complex a setup or something that depends on it, I highly recommending scrapping Sendmail altogether and using postfix. I never had a problem with YAST2 fiddling with the config on that (although once I got the basics set up I abandoned YAST2 for config purposes in favor of emacs)...
[User Picture]
From:adriang
Date:January 9th, 2005 06:05 pm (UTC)
(Link)
I'm one of those people who has actually learned how to change sendmail configuration files, so while I wouldn't recommend sendmail to everyone, I am at least very comfortable with sendmail, myself.

Adrian
From:technoshaman
Date:January 9th, 2005 06:08 pm (UTC)
(Link)
If you can actually grok a sendmail.cf, then postfix will be a walk in the park... albeit quite a different paradigm. It's actually designed to be *easy* to configure. :)
[User Picture]
From:adriang
Date:January 9th, 2005 06:19 pm (UTC)
(Link)
I'm sure you're right. To give you an idea of what you're up against in trying to convince me to chance, I still prefer using 'vi'. I even know how to use 'ed'. 8-)

Adrian
From:technoshaman
Date:January 9th, 2005 06:22 pm (UTC)
(Link)
That's ok, my old boss, who's been doing this since I was in short pants, still prefers a tarted-up version of vi.... but he's learned to love Postfix. I stole my ruleset from him. :)
My Resume Powered by LiveJournal.com