Over in the source repo we have a ticket awaiting moderation (i don't recall that ever happening before!). According to the user permissions docs, anyone with the "moderate ticket" permission (which we derive from "developer") can moderate those. The ticket entry clearly shows it as awaiting moderation, but doesn't seem to offer a way to actually approve/disapprove it:

https://fossil-scm.org/fossil/info/81a5ad0370d9b642

Thus how to handle that ticket, other than through resolving/closing it, is unclear. (The resolution requires someone familiar with the git export who can update the being-ticketed docs appropriately.)

Rather unexpectedly... that ticket is also visible to anyone, even though it's marked as awaiting moderation (just tested from a different browser where i'm not logged in). Feature or bug?

Hello! That was mine. I raised a test bug ("fix a wiki page") and had a confusing but amusing experience.

All the following is with me as a logged-in user.

Step one: find out how to access the ticket system, because I couldn't see any button or link. The method I came up with was to search Google for "site:fossil-scm.org ticket", which then gave me a URL that showed me one single ticket, with an option that said "New Ticket". I now notice that had I used duckduckgo, as I normally would, that same search gives this much more useful URL near the top of the hits.

Step two: get it right first time. Because when I clicked on "Edit" having submitted my ticket, wanting to reduce my wiki page improvement from a critical bug to something a little more realistic, I got the opportunity to edit the subject line, but everything else had been deleted. I didn't carry on editing, who knows what it would put in the ticket database :-)

Step three: Mystery... no way of knowing what happened... none of the links did anything, until I clicked on "status", which said it was awaiting moderation. I have no idea how I did any of the above. Stephan says it is not usual, but I have no idea how to get on the usual track.

I suggest the following might help:

1. Show a link to the ticket system in the UI. Yes there is a potential spam problem, but is it any more so than with the forum?

2. Explain moderation or remove it, as per Stephan Beal's comments here. There is definitely a case where secret/moderated bugs are important, which is for security issues with widely-deployed software. This can be done via moderation, or directing people to email security@, or other means. But I agree with Stephan, it needs discussed.

Dan

Step one: find out how to access the ticket system, because I couldn't see any button or link.

That's intentional. We prefer the forum for that purpose, though developers occasionally (rarely) use it to file tickets.

... but I have no idea how to get on the usual track.

Other than resolving/closing it, i'm not either :/. i just edited the severity via the /reportlist page, but am not sure how to get it out of moderation-pending state. Visiting it via that route does not give any indication that the ticket is pending moderation. (See the bottom of this post.)

Show a link to the ticket system in the UI. Yes there is a potential spam problem, but is it any more so than with the forum?

We don't generally want to encourage people to use the core repo's ticket system. Back when it was opened to the general public (years ago), it seemed that a majority of the traffic was "how do I do X?" and similar questions which were better suited to the mailing list.

Forum spam is surprisingly rare, rarely more than 1-2 attempts/month (usually in bursts of 2-4 attempted posts at a time), and the primary driving factor for the forum was indeed to get away from the ever-more-obnoxious spam which plagued it.

Explain moderation or remove it

Aha, found it: to moderate a ticket you have to visit the artifact, not the ticket. On that page is an approve/disapprove option. Okay, that kinda makes sense but is inconsistent compared to forum moderation, where we visit the forum post to approve it. (It might work through the artifact link - never tried that.)

That's problem one solved. It's still an open question whether it's intended that an in-moderation ticket should be world-readable.

Stephan,

What you said seems to be repeated here in more detail at the SQLite bug reporting page, which explains that:

1. Bug reports are to be made to the Forum first, and
2. The bug database is meant an am important, high-quality historical reference

I suggest that Fossil just copies the SQLite policy on bug reports.

That means Fossil can display the same read-only links to the bug database as SQLite does.

Dan

That means Fossil can display the same read-only links to the bug database as SQLite does.

Except that:

1. It's probably too late for the "high quality historical" part ;). It was open to the general public early on. https://fossil-scm.org/fossil/tktview?name=6b8ee7d40d, https://fossil-scm.org/fossil/tktview?name=d5b429766a

2. We really don't use the core repo's ticket system all that much. If you look at the all-tickets report:

https://fossil-scm.org/fossil/rptview?rn=1

And then sort that by creation date, you'll see a full 5-year (edit: 4.X-ish) gap between Dec. 2015 and Feb. 2020. Several (5?) of the tickets added this year were for tracking issues related to the recent security-relevant releases (and those tickets were initially created in a separate standalone repo and ported back into the main one).

(I don't know how to reply to forum issues by email, and I don't know how to get the web ui to quote the forum issue I'm replying to. So I'm pasting in what my mail client would have quoted :-)

Le mer. 2 sept. 2020 à 17:05, Stephan Beal noreplyca041e47f@fossil-scm.org a écrit :

Forum post by stephan on 2020-09-02 16:04:54 https://fossil-scm.org/forum/forumpost/321e90f0da

That means Fossil can display the same read-only links to the bug database as SQLite does.

Except that:

1. It's probably too late for the "high quality historical" part ;). It was open to the general public early on. https://fossil-scm.org/fossil/tktview?name=6b8ee7d40d, https://fossil-scm.org/fossil/tktview?name=d5b429766a

You're right. So presumably it's ok to zero out the ticket database and start again with a new policy. Those 5 tickets you refer to can be pasted into static wiki pages with explanatory text at the top about how this is a one-off.

2_ We really don't use the core repo's ticket system all that much. If you look at the all-tickets report:

https://fossil-scm.org/fossil/rptview?rn=1

Quite. But Fossil is self-hosting, which maintains the ticket system should be using the ticket system, or not having one at all, because dogfooding has value. Especially since its sister project SQLite has showed the way in how to use the ticket system. So I suggest clearing out the rubbish and starting again, preserving the tiny amount of useful information in it.

We can also improve the process by learning from Github. Developers registered with a project can convert many kinds of object to an issue, and I think it would help populate the high-quality tracker if developers could do something similar for Fossil. From the Github issue creation docs:

You can open a new issue directly from a comment in an issue or a pull request >review. For more information, see "Opening an issue from a comment." If you're using a project board to track and prioritize your work, you can >convert project board notes to issues. For more information, see "About project >boards" and "Adding notes to a project board."

For general users and anonymous users, Fossil can learn from the Githug feature of templates for issues and pull requests. Making a Fossil forum post could remain as it does today by default, just blank. But the UI would also give the user these options to select:

1. Quote the previous message in thread, if any (with the cursor going just under the first quote descriptor, as a nudge to discourage top posting)

2. Functionality question

3. Problem report (notice I didn't say "bug")

4. Enhancement or suggestion

I think we can see from Github that these will help introduce more structure without being an imposition.

And that the "Convert to ticket" facility for authorised users would be a powerful addition to Fossil, likely to quickly start populating the new Fossil ticketing system.

Plus, SQLite should benefit from "convert to ticket", or at least demonstrate my suggestion doesn't work :-)

Dan

(I don't know how to reply to forum issues by email,

You can't. That would be cool at times, but fossil doesn't do that. ("Patches are thoughtfully considered," though.)

and I don't know how to get the web ui to quote the forum...

That's not as trivial of a problem as it initially sounds. i tried to implement that a few months back and fell flat on my face because, in short, the forum supports 3 input formats and at the point we're quoting, we don't know which one the original post used nor which one the response will use, so don't know what is markup and what is content, nor how to escape it for the response.

So presumably it's ok to zero out the ticket database and start again with a new policy.

This is fossil. By design it remembers all mistakes and does not zero out anything unless the data is "shunned," but that's "the nuclear option," not something to be taken lightly.

... because dogfooding has value.

Though the overwhelming majority of the dogfooding happens under this project's domain, sqlite3 does the ticketing dogfooding for us ;). Tickets are actively used/maintained in that project, and these two projects have a recursive relationship.

... and I think it would help populate the high-quality tracker if developers could do something similar for Fossil.

Population doesn't mean much if it's not maintained, and few or none of the active volunteers here have shown an interest in wanting to maintain tickets they aren't themselves actively working participating.

In this particular project, there's generally been not much necessity to maintain tickets. We get a report in the forum and it's either fixed quickly because it's a horrible problem or it scratches one of the developers' itches, or it doesn't get fixed because neither of those applies to it. For better or worse, that's the pattern we collectively tend to follow. There's no "prohibition" or "stigma" against using the ticketing system, we just don't generally tend to do so.

That said: if you're volunteering to become our ticket manager/maintainer, certainly nobody would mind! As Warren famously put it, fossil is a "do-ocracry," so what gets done gets done and what doesn't get done doesn't get done.

I think we can see from Github that these will help introduce more structure without being an imposition.

i agree fundamentally, but am not volunteering for it ;).

And that the "Convert to ticket" facility for authorised users would be a powerful addition to Fossil, likely to quickly start populating the new Fossil ticketing system.

i'm not clear what fossil might have to "convert to ticket," other than a forum post or maybe selected lines of code. In this project and sqlite the sources and forums are separate repos, so "convert forum post to ticket" wouldn't work in these two unless we maintained those tickets in the forum repository.

Quote the previous message in thread, if any (with the cursor going just under the first quote descriptor, as a nudge to discourage top posting)

They also need a nudge to trim their quotes, to avoid a forum full of 100-line quotes with single-line responses. How to quote, however, depends on both the original post's format and the response's chosen dialect, and the response format can change at any time while responding.

If you can genuinely find a good way to do that in our multi-format forum, i'm all ears. That's a feature very dear to my heart, but how it can work when both the quoted message(s) and response can be in any of 3 different text formats is a mystery to me. The near-term plan (as soon as my RSI-ridden hands are up for that task) is to move responding to JS in the client side, so any server-side solution is a dead end. In a single-dialect forum it's an easy problem to solve. It's our multi-dialect feature which makes it difficult.

There was a lot in this reply, thanks Stephan. You pointed out some giant rabbitholes to avoid. I respectfully decline your invitation to be ticketmaster for a project that seems to be very successfully not using ticket tracking. I'd be happy to summarise this and the related matters in the Fossil documentation if that would help others asking the same question.

One tricky question that remains open is this: the Fossil forum uses three non-deterministic formatting layouts, so how can sensible reply quotes be prepared? This matters to you and I suspect many others.

My first contribution is that any email reply feature be considered blocked by this formatting question because the quote formatting ideas of thousand email clients probably makes the question moot forever.

The three formats are: Markdown, Wiki and traditional plain text with multi-indented quotes.

I notice that there was recently a bug in the forum wiki rendering code that wasn't noticed for a long time because wiki use (for this project at least) is rare. However I suspect there isn't any chance of deprecating wikitext for Forum posts because I have read this: https://fossil-scm.org/home/doc/trunk/www/wikitheory.wiki and I can see it is intended to be a globally consistent markup. But if it was possible to deprecate wiki in the forum then that would reduce the problem space.

I don't know how it might help, but I notice RFC7763 defines the MIME type text/markdown and it's also on the IANA list . I'd put this mime type in the "helpful hint but might lead to tears" class.

As a first step, what about adding mime type metadata to all forum posts, inventing the type text/fossilwiki ? After all there is a manual UI selector so why throw away that data?

Dan

what about adding mime type metadata to all forum posts

Forum posts already have a mime type. See https://fossil-scm.org/home/doc/trunk/www/fileformat.wiki#forum

This is so Fossil knows which rendering engine to use.

The problem is how to embed a quote in one format into a post in another format?

One answer is that the auto-quoter force the format of the post.

Another is to invent a special markup for quoting that indicates the format of the quote.

There is precedent for the second option. Fenced code blocks can accept an optional language specifier. Maybe the new quote markup could be a "fenced quote block" with a format specifier.

The problem is how to embed a quote in one format into a post in another format?

That, and especially how to do it when multiple posts are quoted within a response, perhaps at different nesting levels, perhaps with different wiki formats. So we might have a markdown post quoted within a plain text post, which in turn gets re-quoted in a wiki post.

One answer is that the auto-quoter force the format of the post.

If you mean to force the responder to use the format of the post being responded to, that likely wouldn't go over well. It may also interfere with quoting from multiple posts in the same response (which is commonly seen in long threads).

Another is to invent a special markup for quoting that indicates the format of the quote.

There is precedent for the second option. Fenced code blocks can accept an optional language specifier. Maybe the new quote markup could be a "fenced quote block" with a format specifier.

i've tried to reuse the code blocks for essentially that same purpose in some embedded docs and the problem with it is where the CSS class gets assigned. Such a block gets translated (not just in fossil, but in other markdown implementations) to:

<pre><code class="XYZ">...</code></pre>

In terms of CSS selectors, that's the least useful option of them all because CSS doesn't give us a way to say "select the PRE parent of a CODE element with class XYZ" (we can't look up the hierarchy in CSS). Far more useful would be:

<pre class="XYZ"><code>...</code></pre>

With that phrasing we could select and style both the PRE and the CODE item separately because CSS lets us look down the hierarchy. With the current/semi-standard phrasing, we can only style the CODE portion in a language-specific way, but are stuck with the same PRE formatting for all such blocks, regardless of their format specifier. However, that would make ours different than every(?) other markdown formatter, which would be less than ideal.

True, we "could" recognize such blocks in the markdown parser when running in forum mode and treat them differently, but that sounds like a can of worms to me, especially when mixing quotes from different formats.

I notice that there was recently a bug in the forum wiki rendering code that wasn't noticed for a long time because wiki use (for this project at least) is rare.

It was essentially the size which triggered that bug. Had it been a single paragraph, that bug would have continued to go unnoticed. "It might be interesting" to dig through the forum post manifests and measure the usage of each format. Markdown is the clear leader, but whether we have more plain text or fossil wiki is anyone's guess.

However I suspect there isn't any chance of deprecating wikitext for Forum posts because I have read this: https://fossil-scm.org/home/doc/trunk/www/wikitheory.wiki and I can see it is intended to be a globally consistent markup.

Background, for completeness's sake: Fossil's own wiki predates markdown's inclusion in fossil by a good 4-5 years. At the time fossil began development (2007), there wasn't a de facto standard out there. Markdown existed at that time but did not enter really widespread use until github (opened in April 2008) popularized it. It was arguably 2010/11-ish before markdown really started to take the world by storm, at which point we already had a great many wiki pages in fossil's format. The past few years, markdown has become the preferred one within this project (i base that solely on its increased frequency of use compared to fossil wiki), but it would not surprise me if some documentation-only repositories stick with fossil wiki because it's closer to HTML and because fossil supports an option for allowing all HTML (plus fossil-format intra-page links), and thus it's something close to a dialect even many non-technical people can get by with.

But if it was possible to deprecate wiki in the forum then that would reduce the problem space.

That's and idea i could get behind.

As a first step, what about adding mime type metadata to all forum posts, inventing the type text/fossilwiki ? After all there is a manual UI selector so why throw away that data?

For fossil format we use text/x-fossil-wiki. The raw, unparsed posts of course record their own mime type, as it's necessary for rendering, but after rendering to HTML, that type is essentially moot, as we've lost/modified enough of the original formatting at that point that we're then working with plain HTML.

When quoting a post, it would, for basic cases, be possible for the server to quote the original post in full in some halfway useful way only if the server knows the format the responder wants to use. That info "could" be provided to the server when responding, for use in auto-quoting, but...

• The client can change the format at any time while editing, invalidating such quoting.
• Nested quotes from posts with different formats would probably be difficult to get right. (Where "get right" means "quote without losing any formatting/information." Manual copy/paste quoting (what we have now) often loses formatting such as italics or backticks.)

All that said: if we indeed dropped support for fossil wiki format in the forums (keeping it in the wiki and embedded docs), automated quoting would become much simpler. That wouldn't eliminate the problem of needing to know both the source format and response format for proper creation of the quotes, though.

"It might be interesting" to dig through the forum post manifests and measure the usage of each format. Markdown is the clear leader, but whether we have more plain text or fossil wiki is anyone's guess.

I think you are implying Usage Since Epoch as a measure but surely it's about the rate of change and how much wikitext was used in recent times?

But that aside. If you can "get behind deprecating wikitext" what might that mean for quoting on the forum? Some thoughts:

1. Falling back to quoting as it is today is not disastrous, that is, cut/paste. So the new UI quoting feature doesn't have to be prescient, it just has to not be worse than cut/paste. (And it could be worse if the code tries to be too clever.) If there is wikitext markup to be quoted, then we can just say "use the old cut/paste method". We won't always know when there is wikitext to be quoted. But we will at least some of the time and that reduces the problems.

2. The new UI quoting feature can ask for user assistance, just like selecting the markup style today. I think it could be done by having two buttons "Reply Plain Text" and "Reply Markdown", with a checkbox for "Quote". Hopefully a good chunk of the time the user will decide to quote in a way that is most compatible with what is going to be quoted. They won't always know and they won't always get it right. But user input should help reduce the problems on average.

3. Can metadata be added about quoting within a message? I wonder if there is an approximate analogy with RFC2822 headers, which can be pretty reliably threaded despite non-determinism and space-stuffing for quoting in RFC3676.

It's the RFC3676 that gives me pause, because flowed text and quoting are often awkward. Section 3.2 on "Embarrassing Line Wrap" feels pretty relevant and it isn't even dealing with two rendering formats.

Dan

PS the last sentence in the jwz article linked above is "just say no to databases!" :)

I think you are implying Usage Since Epoch as a measure but surely it's about the rate of change and how much wikitext was used in recent times?

The forum's only been online a bit over 2 years, and by that point most of us had become markdown converts (or quickly did so after the forum's launch), so i suspect there's not all that much difference since then. Some difference, but not much (and in markdown's favor).

The new UI quoting feature can ask for user assistance, just like selecting the markup style today. I think it could be done by having two buttons "Reply Plain Text" and "Reply Markdown", with a checkbox for "Quote".

That certainly sounds reasonable.

Can metadata be added about quoting within a message?

Maybe in future messages, but not retroactively.

which can be pretty reliably threaded despite non-determinism and space-stuffing for quoting in RFC3676.

The latter says:

In the presence of these parameters, trailing whitespace is used to indicate flowed lines and a canonical quote indicator is used to indicate quoted lines. This results in an encoding which appears as normal Text/Plain in older implementations, since it is in fact normal Text/Plain, yet provides for superior wrapping/flowing, and quoting.

That sounds more or less like what the markup parser currently does. If you'll look at this response in unformated form you'll see that the above copy/pasted quote spans multiple lines (all but the first indented from the paste) but only has a single leading > to mark it as a quote. The "trailing whitespace" (i.e. all following lines up to the next run of two adjacent newlines) is used to determine that it belongs to a single quote.

It's the RFC3676 that gives me pause, because flowed text and quoting are often awkward. Section 3.2 on "Embarrassing Line Wrap" feels pretty relevant and it isn't even dealing with two rendering formats

Deeply-nested quotes are an unavoidable fact of life but, interestingly, seem to be non-existent in this forum (very probably because we don't auto-quote!). Nesting of deeper than 2 levels is currently unheard of around here, but could certainly become a reality if responses included auto-quoting.

PS the last sentence in the jwz article linked above is "just say no to databases!" :)

It's also from 2002, before sqlite had taken the world by storm ;). i seem to recall reading his name years ago in conjunction with xemacs?

In any case, this is all stuff i'll keep in mind when ajaxifying the forum, but i'm not optimistic about being able to pull off a sensible quote mechanism except via an approach like your suggestion of a "quoted reply in FORMAT X" button. In such a case, the server has access to both the post being quoted (so it has the unparsed bits in its original format) and the response format, so it could hypothetically escape the quoted bits so that they won't be mangled in the response.

very probably because we don't auto-quote

Which, incidentally, also prevents top-posting with a dozen full-text replies below that, even though every mail client worth talking about has supported threading in some form for decades, so there's zero point in preserving prior replies in full.

That's also why no one's ever done more than 2-level manual quoting here, that I recall: Fossil's blockchain data structure durably maintains the links between posts, so you simply cannot cause a reply to become unlinked from its parent. All you need in quoting is a hint about what you're replying to.

(With email, you can at least make the argument that you need full-quoted top-posting in the case of forwarded messages, but that can't happen here, so it's an irrelevant use case to the current discussion. The closest equivalent would be emailing a link to a thread, which then gets you the full context.)

A single level is only needed when the parent has multiple points, and you want to respond to it piecemeal, even incompletely, so a bare reply might be confusing.

I've only used 2 levels of quoting when the sub-threads get complicated and it's helpful to do a "and then I said, and then you said, and so here's my answer" kind of thing.

I'm struggling to imagine why I'd need a third level.

...jwz...

He of Zawinski's Law, "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can."

Fossil can read mail, and Git cannot, so Fossil wins. Obviously. /s

very probably because we don't auto-quote

Which, incidentally, also prevents top-posting with a dozen full-text replies below that

How about semi-auto quota on in Javascript?

I recently saw, in the Discourse forum software, a feature where you select some text, then a "Quote" button appears. When clicked, a quote block with the selected text is inserted at the current local in the text entry area.

A Fossil version could insert either a HTML or plain text quote block.

Deeply-nested quotes are an unavoidable fact of life but, interestingly, seem to be non-existent in this forum (very probably because we don't auto-quote!). Nesting of deeper than 2 levels is currently unheard of around here, but could certainly become a reality if responses included auto-quoting.

That's a really interesting observation and it sounds like a good way of keeping communication clean, with users manually refreshing the context of a thread.

How about auto-quote does two levels, and the "Quote all levels" button needs to be pressed if you really want all of them. This also reduces the volume of the quoting problem in general. More importantly, and like the "Quote in Format X" button previously discussed, it also gives data about how users really want to use these features.

Dan