Warren Young (wyoung) on 2020-10-25 14:49:39:

Concerning inbound SMTP:

does Fossil include an SMTP server?

Not one that's fully functional and useful on the modern Internet. You'd have to ask drh what his plans are for bringing it to a state where it's functional enough that it can successfully deliver mail to N-nines percent of the existing Internet SMTP infrastructure. It would also be interesting to know his value for N.

The architectural question that helps here is: "who is doing the queue management?"

If it is Fossil, then as you imply Fossil needs to maintain a full-service internet-safe smtpd, which brings up hard questions including how many nines, spam, which of the dozens of RFCs will be supported, and security. This is a large and ambitious project.

If the answer is "mail queues get handled outside Fossil" then that's what the LMTP protocol in RFC2033 is for, in fact it states that implementations MUST NOT listen on port 25. Dovecot's LMTP server can also act as an LMTP proxy, so it is useful code to study for Fossil's potential needs for input and output. LMTP is a subset of SMTP, so the existing code in Fossil is well on the way. Postfix, Cyrus and OpenSMTPD among others also have LMTP code to read.

Dan Shearer

@drh, would you mind commenting on your views of/intentions for the Fossil smtpd?

I'm asking now because I included my take above in the example GSoC project areas and I think it could be a good one. However you must have had something in your mind? I don't want to propose something that you are against in principle.

It is fair to say I (and seemingly Warren) are quite sceptical of the notion of Fossil including an internet-facing replacement for Postfix/etc. Writing an SMTP daemon isn't the hard bit, it's keeping maintenance sane in the face of ever-changing insanity. It's certainly doable but I'm wondering why.

I had hoped the LMTP idea might be a good compromise to deliver (ho ho) what you were looking for without the open-ended commitment to a mail server. Without in any way suggesting that world doesn't need a better free smtpd. But Fossil isn't that.

Dan Shearer

It is fair to say I (and seemingly Warren) are quite sceptical of the notion of Fossil including an internet-facing replacement for Postfix/etc

FWIW, i'm 100% skeptical of fossil hosting any sort of mail infrastructure beyond outbound-only alert notifications (while also noting that that limitation makes it impossible for fossil to automatically handle email bounces). That seems so very far out of scope to me and introduces excessive complexity and potential new attack vectors.

i haven't looked closely at LMTP, so cannot comment on it (and the linked-to page about it jumps right into the config file rather than telling us what it is), but the "MTP" part of the name is already enough similarity with SMTP to give me the willies.

0.02€ and all that.

I was experimenting with that at about the same time the Forum was coming on line. I was thinking, for a while, that Fossil would host an ordinary mailing list. It was also an attempt to break dependency on gmail. Those goals have mostly been abandoned at this point.

It would still be good for Fossil to be able to send and receive its own email, so that it could do email notifications without having to rely on an external SMTP process. For now, I'm not pursuing that goal. The old SMTP work is a dead experiment.

Even if we do continue to require an external SMTP process for email notification, we do need to enhance Fossil so that it can at least understand email bounces, and disable email notifications to addresses that bounce too much. That is an important deficiency in the current email notification system.

I have two ideas for handling inbound email.

First, add an imap or pop client to fossil and run it from the backoffice. To set up email integration, configure the MTA to save the bounce emails in a mailbox. Fossil can use pop or imap for receiving the emails and process them.

Advantage: fossil can run on a separate host from the email host. Pop/imap are supported by Exchange, gmail and implementations exist for Unix MTA's.

Disadvantage: needs backoffice which IIRC has had issues in the past on bsd and windows boxes.

A second way to do it is to add a command to fossil:

  fossil email -R repo receive_bounce

Set up the MTA to pipe the bounce message into this fossil command. Advantage: MTA triggers the import of email on demand. Disadvantage: Fossil needs to run on the same host as the MTA. (This could be exposed as /email/receive_bounce with a simple client that pipes the message over the network to the web site.) Using Exchange or other windows MTA may not support running a program, Gmail certainly won't.

Both of these could be extended to act as a forum<->email gateway. Option 1 would poll a forum email box/address (rather than the bounce address). Option 2 would use the fossil email -R repo receive_post command.

Thoughts?

A second way to do it is to add a command to fossil:

fossil email -R repo receive_bounce

This is better, at least for start, because with this and some external tools (like getmail or fdm) one can script the first proposal quite easily. Moreover when all the email-processing procedures will be ready, adding POP/IMAP client on top of it would be much easier I suppose. Just my $0.03. ;-)

John Rouillard (rouilj) wrote on 2021-02-17 20:59:21:

A second way to do it is to add a command to fossil:

fossil email -R repo receive_bounce

... etc

It would help if we use the terms defined in RFC 5598:

• mail Transfer, the "T" in SMTP and LMTP
• mail Delivery, which is what a Mail Delivery Agent (MDA) or a Local Delivery agent does . A delivery agent may use LMTP.

In these terms:

• drh proposed implementing an MTA. That doesn't seem necessary now, is hard to make safe, reliable and multiplatform. I also really dislike the idea
• rouilj proposed implementing a non-network-aware Mail Delivery Agent above, where its security policies would reject anything that doesn't look like a bounce it is expecting

The proposed Mail Delivery Agent is exactly what existing MTAs expect to see (Postfix, Dovecot, Courier, Exim, etc). And the fact that it is non-network aware is the kind of problem mail administrators expect to deal with. There is nothing special about that.

The above is all definitely true for Unix, and only potentially true for non-Unix. I feel there is no point in this forum of worrying about the non-Unix case because that requires so much platform-specific knowledge and setup it is far out of scope for Fossil when we don't even have the ability to deal with a bounce yet.

Dan Shearer