My e-mail client usually threads the e-mail notifications I receive from the Fossil forum, but some of them end up in their own three of messages, detached from the rest of the thread.

For example:

• Post 4, Message-Id: <25b84de0ae5deeedfbc237bd8e79a0f3@fossil-scm.org>

• Post 5, which is clearly a reply to 4, but In-Reply-To: <f48aa988837570820e8f3ae4431adffe@fossil-scm.org> references a Message-Id that my e-mail client has never seen.

This is probably caused by messages being replied to having been edited after the e-mail notification is sent, causing the In-Reply-To header to be generated from a different hash. Is it possible to retain the In-Reply-To of the original message in such cases?

+1. Threading is often broken indeed. It's inconvenient to ML users. (FWIW)

Edit: removed coarse language

Please report whether or not the In-Reply-To for this post is set appropriately.

Yes, thank you!

For me it looks ok (one single thread, six e-mails) both in GMail and in Thunderbird.

That's Thread#2, i.e. the one with 6 posts. I have it too.

But I have that earlier Thread#1 with 2 Lidor posts, on their own.

Also, there's a setting to get email notifs for Edits as well, could
be the difference between you and me here.

There is a new patch for this at check-in fb4545ef937e06ee.

Problem Analysis

Many (most?) users do not receive email notification for edits to prior posts. As each edit has a different message-ID, if you do not receive edit notification, and a subsequent post refers to an edit, this will appear as a broken thread.

To work around this, when a post is a reply to an edited post, the In-Reply-To: field is adjust to point not to the edited post, but to the original version of the edited post.

Before any of the fixes to this, the message-ID used for the In-Reply-To was just the artifact hash of the post that was actually being replied to:

• Original code: https://fossil-scm.org/fossil/info/4c82b8ecaadf2451?ln=2219

My initial attempt to fix this looked followed the "firt" column of the "forumpost" table back to the previous edit. I mistakenly believed that "firt" pointed back to the original content. But that is not correct - it refers to the previous edit only. The first fix was like this:

• First fix attempt: https://fossil-scm.org/fossil/artifact/1a5e4160b687?ln=2219-2221

The latest fix follows the chain of edits back to the original post, using two common-table-expressions:

• Second fix attempt: https://fossil-scm.org/fossil/artifact/34b1f0fb99cc?ln=2219-2229

This is still quite efficient, it turns out.

Here you go, details, since my report was questioned.
We can clearly see different In-Reply-To headers.
And somehow Lidor posts in Thread#1 are older than his Thread#2 one.

Hmmm, I notice Thread#1 has IPv4 addresses for SQLite.org, while
Thread#2 has IPv6 addresses. Normal? The cause for this?

Thread#1, Post#1

Return-Path: <root@sqlite.org>
Received: from sqlite.org (sqlite.org. [45.33.6.223]) […]
        Thu, 16 Apr 2020 05:00:06 -0700 (PDT)
Message-Id: <7a63830292ca0e62600eddb00eb5b7b8@sqlite.org>
In-Reply-To: <dda18b1bcb4c18cc4342227f02e21ab5@sqlite.org>
From: Lidor <noreply5f7ab08f7@sqlite.org>
Sender: <sqlite-forum@sqlite.org>

Forum post by PrototypeX on 2020-04-16 11:57:10
https://sqlite.org/forum/forumpost/7a63830292

Thread#1, Post#2

Received: from sqlite.org (sqlite.org. [45.33.6.223]) […]
        Thu, 16 Apr 2020 05:00:06 -0700 (PDT)
Message-Id: <1054343e86f27f9b36aed9ea147e2b67@sqlite.org>
In-Reply-To: <dda18b1bcb4c18cc4342227f02e21ab5@sqlite.org>
From: Lidor <noreply5f7ab08f7@sqlite.org>

Forum post by PrototypeX on 2020-04-16 11:56:22
https://sqlite.org/forum/forumpost/1054343e86

Thread#2, Post#1

Received: from sqlite.org (sqlite.org. [2600:3c00::f03c:91ff:fe96:b959]) [...]
        Thu, 16 Apr 2020 02:56:18 -0700 (PDT)
Message-Id: <183697d77e4645805c091a7133879f6e@sqlite.org>
From: Lidor <noreply5f7ab08f7@sqlite.org>
Sender: <sqlite-forum@sqlite.org>

Forum post by PrototypeX on 2020-04-16 07:54:40
https://sqlite.org/forum/forumpost/183697d77e

Thread#2, Post#2

Received: from sqlite.org (sqlite.org. [2600:3c00::f03c:91ff:fe96:b959]) […]
        Thu, 16 Apr 2020 03:44:44 -0700 (PDT)
Message-Id: <53951ba6767046979b75c8ab9fc3bacb@sqlite.org>
In-Reply-To: <183697d77e4645805c091a7133879f6e@sqlite.org>
From: Gunter Hick <noreplyb9fdb4a4b@sqlite.org>
Sender: <sqlite-forum@sqlite.org>

Forum post by gunter_hick on 2020-04-16 10:43:42
https://sqlite.org/forum/forumpost/53951ba676

I think what happened here is that the post that linked thread#1 into thread#2 was rejected by the moderators, specifically because it was empty. So this exchange really did get broken up into two separate threads. So, I think the In-Reply-To headers are correct. The problem is that we allowed a thread to be broken in two by moderation.

I'll try to address this. But it seems like a low priority. Potential changes:

1. Disallow new thread posts with empty message. An edit to an existing post with empty content is still ok, because that is how a post gets deleted. But replies with empty content will be rejected. A possible fix for this is no on-line as check-in 3684c58b4b3663b2.

2. Fix the hierarchical thread viewer so that it shows posts that have a missing in-reply-to entry, as missing posts still might happen due to synchronization delays and/or shunning. Notice that the two replies from PrototypeX show up in the chronological view but not in the hierarchical view. Possible fix for this in check-in bed0dd1a956c0887

3. Fix the administrative log to record the hash of rejected posts. It currently only records the BLOB.RID value, which is mostly useless for forensic analysis since the BLOB entry in question will have been deleted.

4. Do not allow a Reply-To for posts that are pending moderation, even by the user or made the original post. This seems to have never been allowed in the first place.

(Edit-1: Better hyperlinks)
(Edit-2: Proposed changes #1 is now implemented)
(Edit-3: Mark off changes #2 and #4)

I think what happened here is that the post that linked thread#1 into thread#2 was rejected by the moderators, specifically because it was empty.

A related problem which has come up a couple times, which might or might not lead to thread breakage: someone creates a post and wants to edit it, but instead (for whatever reason) creates a new post with nearly-identical content. In such cases i've been known to approve the newest one and disapprove the other. In one recent case i approved the older one because i hadn't yet seen the newer one, then rejected the newer one to avoid an almost duplicate post.

Another split thread.

Again, I notice IPv4 versus IPv6 addresses.
So doesn't seem related to moderation, is it?

Any chance sqlite.org would use a consistent IP, to make thread-splitting in GMail client go away?

Thread#1, with 15 posts, starts with

Received: from sqlite.org (sqlite.org. [45.33.6.223])
        Fri, 17 Apr 2020 05:29:52 -0700 (PDT)
Subject: [fossil-forum] Location of .fossil
Message-Id: <c000a7352157276e6be129f8bdc7d647@fossil-scm.org>
From: anonymous <noreply9e9eb482a@fossil-scm.org>
Sender: <fossil-forum@fossil-scm.org>

Thread#2, with 5 posts, starts with

Received: from sqlite.org (sqlite.org. [2600:3c00::f03c:91ff:fe96:b959])
        Sun, 19 Apr 2020 08:52:39 -0700 (PDT)
Subject: [fossil-forum] Location of .fossil
Message-Id: <b6bb01e9e0a1bd0257c8962b4e8cbc4b@fossil-scm.org>
In-Reply-To: <88aea61019104570a652e804783d8e37@fossil-scm.org>
From: Richard Hipp <noreplyce749ed72@fossil-scm.org>
Sender: <fossil-forum@fossil-scm.org>

https://sqlite.org/forum/forumpost/90ddf0971f is split in GMail for me.
The thread title appears to be have been renamed/changed. Can that
explain the split thread? Thanks, --DD

Doubtful. The message-ids are based on the SHA3 artifact hashes of the forum post artifacts. Title changes should not matter.

Perhaps you could figure out where the thread split occurs in your email reader and post the relevant message-ids and in-reply-to fields.

Here are the full details. The In-Reply-To headers appear correct.
It's probably GMail that decides the Subject header change is reason
enough to split the thread. Sorry for the noise... --DD

PS: Unless Fossil forces email notifs to use the original Subject
in all thread-related notifs, to avoid breaking the thread in GMail.

Here's the original post, in its own thread (just that message):

Received: from sqlite.org (sqlite.org. [45.33.6.223])
        by mx.google.com with ESMTP id ...
        Mon, 27 Apr 2020 03:30:38 -0700 (PDT)
Subject: [sqlite-forum] SQLite crash (3.31.0-3.31.1)
Message-Id: <90ddf0971f3625bed261f39e535d9271@sqlite.org>
From: Gabriele <noreply88f69cfcb@sqlite.org>
Sender: <sqlite-forum@sqlite.org>
Date: Mon, 27 Apr 2020 10:30:35 +0000
...
Forum post by aeroxr1 on 2020-04-27 10:28:33
https://sqlite.org/forum/forumpost/90ddf0971f

Here's the second thread's 1st message, with 5 messages (last from Dan):

Received: from sqlite.org (sqlite.org. [45.33.6.223])
        by mx.google.com with ESMTP id ...
        Mon, 27 Apr 2020 18:30:29 -0700 (PDT)
Subject: [sqlite-forum] SQLite Crash: EXC_BAD_ACCESS (code=1, address=0x54)
Message-Id: <2f1ffbe2ef3be7eb4266275fe02d24bc@sqlite.org>
In-Reply-To: <90ddf0971f3625bed261f39e535d9271@sqlite.org>
From: Larry Brasfield <noreplyf16e6c3f5@sqlite.org>
Sender: <sqlite-forum@sqlite.org>
Date: Tue, 28 Apr 2020 01:30:20 +0000
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Forum post by LarryBrasfield on 2020-04-28 01:30:16
https://sqlite.org/forum/forumpost/2f1ffbe2ef

Here's the third thread (one more since last reported) sole message:

Received: from sqlite.org (sqlite.org. [45.33.6.223])
        by mx.google.com with ESMTP id ...
        Tue, 28 Apr 2020 04:57:36 -0700 (PDT)
Subject: [sqlite-forum] BUG - SQLite Crash: EXC_BAD_ACCESS (code=1, address=0x54)
Message-Id: <20296cc44f52d607a08c46f3b2910687@sqlite.org>
In-Reply-To: <b606a4ce88ccc81c268c03031c552aae@sqlite.org>
From: Gabriele <noreply88f69cfcb@sqlite.org>
Sender: <sqlite-forum@sqlite.org>
Date: Tue, 28 Apr 2020 11:57:26 +0000

Forum post by aeroxr1 on 2020-04-28 11:55:29
https://sqlite.org/forum/forumpost/20296cc44f

which seems to thread correctly with Dan's message in the 2nd thread:

Received: from sqlite.org (sqlite.org. [45.33.6.223])
        by mx.google.com with ESMTP id ...
        Tue, 28 Apr 2020 04:40:33 -0700 (PDT)
Subject: [sqlite-forum] SQLite Crash: EXC_BAD_ACCESS (code=1, address=0x54)
Message-Id: <b606a4ce88ccc81c268c03031c552aae@sqlite.org>
In-Reply-To: <236a1864c39edba3985673068380441d@sqlite.org>
From: Dan Kennedy <noreplybf9ccc888@sqlite.org>
Sender: <sqlite-forum@sqlite.org>
Date: Tue, 28 Apr 2020 11:40:27 +0000

Forum post by dan on 2020-04-28 11:39:30
https://sqlite.org/forum/forumpost/b606a4ce88