Date: 2003-12-01 07:31:00
Tags: email, unix
qmail may be eating your email
The other day I placed an order online, and entered my email address to get a confirmation message. It was supposed to be sent right away after placing the order, but it didn't arrive. However, since I have been watching my mail server logs more closely recently (due to greylisting), I noticed that the company's mail server had attempted to deliver my confirmation message several times unsuccessfully. I found that this had nothing to do with greylisting rejecting the message, it was being passed straight through to qmail.

The qmail logs were showing an incoming connection but no message was logged as being delivered. Some sleuthing around with the help of davehart, including writing a SMTP shim that logged the precise contents of the mail delivery transaction, revealed the problem. Qmail was rejecting the message with an error message: "451 See http://pobox.com/~djb/docs/smtplf.html."

It turns out that the sending software was trying to send a message that was composed with only LF characters at the end of each line (normally email is sent with both CR and LF line ending characters). Qmail considers this illegal, and when it detects such characters it responds with the above error message and immediately drops the connection. This causes the sending server to save the undelivered message and try again later.

The qmail document refers to a document it calls "822bis", which apparently was an older name for RFC 2822. Section 2.3 of this document does indeed prohibit the use of bare LF characters within the message body. However, this document is not considered a standard; to find a standard relating to email transfer we must look at RFC 822, also known as STD 11. RFC 822 does not specifically prohibit the use of bare LF characters in message bodies.

RFC 821 also requires, in section 4.1.1, that "The receiver should not close the transmission channel until it receives and replies to a QUIT command (even if there was an error)." Based on this, I have concluded that the qmail behaviour is in error and should not be used as is.

The culprit in qmail is the function blast() in the qmail-smtpd.c file. This function scans the message body and if it detects a bare LF character, calls the straynewline() function. This function prints the above 451 message, and abruptly exits:
void straynewline() { out("451 See http://pobox.com/~djb/docs/smtplf.html.\r\n"); flush(); _exit(1); }

There is no evidence that anything bad will happen if the bare LF characters are permitted in the message body. So, in order to fix this problem in qmail, I modified qmail-smtpd.c so that the straynewline() function takes no action:
void straynewline() { }

This modification finally let the order confirmation I was expecting arrive successfully.

I recommend that everybody who runs qmail apply this modification. You may be missing mail and not know it.
[info]dbaker
2003-12-01T05:35:18Z
This was a big issue in 1997 or so when several major MUAs (Eudora?) would generate message with bare LF characters.
[info]ghewgill
2003-12-01T05:38:13Z
Sure, but as long as the MTA ends the SMTP body with CR LF . CR LF as required, it shouldn't matter what the MUA creates.
[info]dbaker
2003-12-01T05:41:02Z
Messages are transmitted from a MUA to a MTA via SMTP.
[info]ghewgill
2003-12-01T05:42:02Z
Oh uh yeah, good point. :)
[info]nugget
2003-12-01T05:38:52Z
More and more I am finding myself drawn to investigating postfix. Aside from being fond of dot-qmail(5) files and feeling secure I suspect that I'd benifit from the migration.
[info]pasketti
2003-12-01T07:49:07Z
When I upgraded my Linux box a few months back, I took the opportunity to switch to postfix.

So far, it's been great and has given me no problems at all. I'm currently looking into implementing greylisting for it.

[info]dbaker
2003-12-01T08:43:11Z
What's great about it?
[info]pasketti
2003-12-01T12:21:21Z
I switched from sendmail, so just about everything.

The best thing is that you can read the config files.

It's also fast. It will send out messages in parallel, which isn't that big a deal unless you're running mailing lists.

[info]decibel45
2003-12-02T11:20:47Z
Not having to deal with crap like this? :)

The only downside I've seen to postfix is that (last I looked), you couldn't send all email for an entire domain to one email address with a single alias like you can in qmail.
[info]luckman
2003-12-01T20:32:03Z
Well I'll be a sonofabitch.
[info]davesill : qmail is doing the right thing
2004-02-19T14:31:17Z
Don't blame qmail for doing the right thing. Blame the broken client.

See Jonathan de Boyne Pollard's "qmail Myths Dispelled" for a good analysis of the bare LF problem and why Postfix's behavior is harmful to interoperability.
Greg Hewgill <greg@hewgill.com>