Fossil Forum

Forum Enhancement Wish-List
Login

Forum Enhancement Wish-List

Forum Enhancement Wish-List

(1) By Richard Hipp (drh) on 2019-11-27 19:56:53 [source]

The Forum has been in operation for over 15 months now. I am very pleased with it. I think it works much better than a mailing list. But, it could still be improved. In this thread, I propose several suggested improvements.

Reply on this thread with refinements or questions or dissent concerning these suggestions, or to make new suggestions. Feel free to take up any of these ideas and work on them, if they appeal to you.

(1) In-Thread Edit And Reply

As currently implemented, pressing the Edit or Reply buttons on a Forum post takes you to a separate page with a HTML form that needs to be filled out with the new or revised post. That form page has a "Preview" button to show how the new post would be rendered, prior to display.

I think it would be better if the Edit/Reply buttons used javascript to open up the editor form directly on the thread display page, and that the Preview button added preview text directly to the thread display page using XMLHttpRequest. Preview should be required before Submit, after any changes in the entry box.

(2) Delta Compression Of Edited Forum Posts

Fossil is not currently using delta compression to minimize the size of forum posts that have been edited. It should.

This is an implementation detail, not a user-visible change.

(3) Child Threads

Often a forum discussion about one topic will digress into another topic. When that happens, it would be useful to be able to fork the thread so that the new topic has its own thread. The parent thread would link to the child at the point of the fork, and the child thread would link back to the parent.

Sometimes we do not recognize the need for a child thread until after the discussion has digressed. For that reason, it seems important to be able to create the child thread after one or more posts of the child have already entered the block-chain. This could be accomplished by adding tags to the preexisting posts that change their "thread root". This would be similar to the use of tags to change check-in comments, users, dates, and even parents for check-ins.

(4) Built-in Support For PIC and EQN

I'd like to be able to include diagrams described by PIC and mathematical equations set by EQN in the middle of forum posts (and also in Wiki and Tickets and so forth). That has been discussed before.

This idea is not exclusive to the forum. It also serves the needs of wiki and tickets and embedded documentation.

(5) Finer-grained notification

Add the ability to get email notification to changes to a specific thread, or replies to a specific post. Currently, email notification is limited to notification of any forum post edit, change, or addition.

(6) Forum Threads Rooted On Wiki Pages or Tickets or Check-ins

Wouldn't it be cool to be able to start or contribute to a discussion thread associated with a particular check-in or wiki pages or ticket or feature-branch?

This would allow the wiki to serve as kind of a Blog. The owner would be the only one with permission to edit wiki, and hence the only one who could add or revise the blog entries. But readers could leave comments as part of a forum thread, by giving the nobody/unknown user the ability to add to a forum thread.

A forum thread associated with a feature-branch would be a useful place for developers to discuss the progress or objectives of the branch. Actually, it might be useful to be able to attach multiple forum threads to a single feature branch, as I can envision separate threads for different issues like UX, testing, performance, and so forth.

(2) By Stephan Beal (stephan) on 2019-11-27 20:09:41 in reply to 1 [link] [source]

(6) Forum Threads Rooted On Wiki Pages or Tickets or Check-ins

That is an interesting idea but assumes that the project source code and forum are in the same repo instance, which isn't the case for this project (and arguably probably shouldn't be for most high-activity projects).

(3.1) By Richard Hipp (drh) on 2019-11-27 20:26:39 edited from 3.0 in reply to 2 [link] [source]

I kept the Fossil Forum in a separate repo from the Fossil sources, so that people do not have to sync all of the forum history in order to get at the source code.

But, if we were to implement my proposed refactor of the sync protocol, then users would be able to separately clone/sync the sources, the tickets, the wiki, and/or the forum history, in any combination they want. In that case it would make more sense to combine the forum and the sources into the same repo.

Edit: Or... one could host the general user-help forum in a separate repository, but still provide a forum on the source-code repository for use between developers discussing project implementation details.

(5) By Stephan Beal (stephan) on 2019-11-27 21:13:06 in reply to 3.1 [link] [source]

We already have the convention of naming wiki pages for associating them with branches, commits, and tags. Might it suffice to apply the same convention to forum post subject lines? So a subject line of branch/foo would cause that forum thread to appear in the same places a wiki page with the same name would embed?

Interestingly, Andy Goth has kinda/indirectly already done this:

https://fossil-scm.org/fossil/timeline?r=andygoth-restore-related

That branch-associated wiki page simply directs the reader to a forum thread.

On the other hand, this approach might lead to havoc if an ill-behaved user creates a forum post titled branch/trunk.

(4) By Richard Hipp (drh) on 2019-11-27 21:01:53 in reply to 1 [link] [source]

(7) Attachments On Forum Posts

Sometimes it is useful to be able to attach binary artifacts (for example, JPG or PNG files) to a forum post, and in the case of images, to display the image in-line.

This should be a feature that administrators can turn on and off, or perhaps only provide to a trusted core user set. One should also be able to limit the size of attachments.

(6) By SMAH on 2019-11-28 16:55:21 in reply to 1 [link] [source]

I know this is a long wish list. It may be a case of allowing the possibility of adding them in future or now by user customisation (similar to ticket customisation), while maintaining backward compatibility of existing fossil database files.

I would like to see the following:

  1. Add a fossil command line command to create threads and posts - this would be useful for auto-generating repeating threads using cron. Autogenerated threads can be used for example a thread titled "weekly progress discussion for 25/5/2019". Command line generated posts (replying to thread level) can be useful for standard or time triggered notifications - for example "This thread will be locked on 30/7/2019 at 00:15 hrs UTC". It would be useful if an option existed to set the colour of machine generated threads and posts for clarity (so non replying posts are clear) using a flag in the commandline.

  2. Allow multiple forum topics. One big list for all threads is difficult to keep track of. I presume that the best way of implementing this is to add an extra forum topic field to forum threads, and display them based on this topic being selected.

  3. Add the ability by the thread moderator or thread creator to set a list/group of users who are able to post on each thread, and by the moderator to set a list/group of users to whom the thread will be visible. The thread moderator/creator here should be the one who is set as moderator for the thread at its creation and the creator is the person who created the thread (not ones defined for the whole repository) and will be different for each thread. At the moment you get a single massive list of threads, and possibly posts appearing, which can be excessive to keep track of or moderate. The idea behind this is to allow restricting of posting or seeing threads to those who should actually be part of the discussions. Also on fully open forums, the ability to set a different colour on posts for users to be notified, and by group might be useful.

  4. Add locking of threads by their moderator, creator or automated command line scripts so no more posts can be made. The reason for this is that the forum thread list goes on for ever, and it is difficult to find threads and easy to post to old redundant threads by mistake. Three options for locking should be available - lock immediately, lock at date time, lock after a certain time from now. If a lock is set on a thread is set, the type of lock, and the locking date should be displayed at the top of the thread, and locked threads displayed in a different colour, with the selectable ability to list only open threads.

  5. Add the ability to set vote count function (which is recorded in the database) on threads where those registered to post, or registered users, or everybody who can view the page can vote for options (log connection IP address for anonymous users) . The best way to implement this would be in a separate (new) database table containing the voting forms and a way of embedding voting form(s) and their results to threads and posts. The query function results extracted from the count would be: a) boolean - did all registered to vote on this thread respond to the vote form? b) boolean - did none registered to vote on this thread respond to the vote form? c) integer - aggregate number of votes/scores for each option. d) floating point - percentage of aggregate votes/scores for each option. This feature is useful for getting confirmation on any objections/agreements/issues/confirmation that thread has been read etc. at the end of a discussion, and having this recorded in the database. There should be a time when the start of voting should be triggered - typically immediately, by a certain date, after a certain period of time from the thread/post which the voting form is associated is created, or on locking of the thread to which the voting form is attached. Similarly there should be a time after which voting is locked - typically by a certain date, after a certain period of time after the creation of the thread/post with which it is associated, or on locking of the thread with which the voting form is associated. In addition, there should be a way of auto-triggering locking of the thread when a specified vote count is reached eg. when all users on the voting list have voted yes, when any of them have voted no, or when a certain proportion or number of votes for a choice has been reached.

  6. Allow WYSIWIG (most posts aren't edited subsequently), file attachments and easy linking of artifact/wiki/ticket etc. refs to threads/posts on forums (the more you encourage artifact linking in threads/posts, the easier it is to use forums for detailed discussions of specific things - a frame for navigating to the artifacts/wikis/tickets etc. for viewing/linking may be useful for this (the same would also apply to ease of linking of artifacts/wikis/tickets etc. in Wikis)).

(7) By Warren Young (wyetr) on 2019-11-29 19:57:11 in reply to 6 [link] [source]

set the colour of machine generated threads and posts for clarity (so non replying posts are clear) using a flag in the commandline.

Not the color, the CSS class. After that, it's a skinning issue.

Put another way, changing the skin should change the color of non-replying posts as well, so they fit in with the new skin.

multiple forum topics

Back when we were discussing the creation of the forum feature, I asked for the same thing, but now having seen one-forum-per-repo in action, I wonder if this is a reactionary wish for a return to vBulletin/phpBB, Usenet, and FidoNet.

If you have multiple independent, non-overlapping topics of discussion, then do you not also probably need multiple Fossil repos to track the underlying artifacts under discussion?

In what meaningful way is this feature not functionally equivalent to the following?

 $ fossil serve --repolist /path/to/forum/repos

To anticipate one possible rejection of this idea, you might not like the idea of administering the same set of logins across repos, but Fossil already provides a way around that today with login groups.

Add the ability by the thread moderator or thread creator to set a list/group of users who are able to post on each thread, and by the moderator to set a list/group of users to whom the thread will be visible.

This falls out of the prior item. If you have a need for private forums, you can get that today by having multiple repos, one per forum, each with a distinct set of users, each with different user capabilities, so that not all users can participate on all forums.

Inversely, if you had this ability, how would you get it without greatly complicating the user capability system? Be concrete: propose a design.

which can be excessive to keep track of or moderate.

Which Fossil forum is bigger than this one? As one of its small number of moderators, I would not describe the duties of my job here as "excessive."

I'm presuming that you can point to a public Fossil forum, since in a private repo, what you need is not moderators but a culture of social norms that avoid the need for moderators. Put another way, moderation and the org chart are functionally overlapping concepts.

If you're asking instead for Fossil's user capability system to become flexible enough to model your employer's org chart, I don't see that happening.

vote count function

Yes, even to the point f gamification. Adding a +1 to a post is better than the noise replies we're relegated to now. "Thanks!" or "Yes, me too." etc.

There are good models to follow here. Especially for a forum like this one, I think Stack Overflow's model would work well, where a post that's basically a question ends up with competing answers that are voted up by the readers, so that later searchers can quickly find the best answers.

Anonymous voting should be allowed, under moderator control. As a mod, you'd see something like "Anonymous user at 1.2.3.4 voted +1 on this. [Accept] / [Reject]"

There should be a time when the start of voting should be triggered

Why? All that does is push the thread out of people's minds and then force those interested in voting to re-visit the thread later and re-read what's been said so they can cast their vote. That biases votes toward interested parties, which turns it into a political exercise.

The Stack Overflow model is better. Of these choices, which are good options? Those posts that gather the most upvotes are probably better answers than those that did not.

there should be a time after which voting is locked

Under what conditions is that a good idea?

In a technical forum, which will probably dominate the use of Fossil forums for fairly obvious reasons, tech changes, so the "right" answers may also change over time. Why would you want a good answer posted two years after the thread was created to be unable to garner votes?

I've got answers on Stack Overflow that were voted up to the top answer on a topic despite being posted years after the prior top answer, usually because the old answers are just wrong now, and my late answer is now the correct answer.

And vice versa, probably, too, though I'm less likely to notice, since it means I'm now getting fewer up-votes than before. Until someone feels strongly enough to take the penalty to downvote an answer of mine, I may never notice that my old answer has now been outstripped by a newer better one.

when all users on the voting list have voted

That seems easy to script once we have command-line thread locking, not something we should put in Fossil core.

However, I don't really see the point. If the DB keeps track of who has voted and prevents multiple votes, then what's the point of locking the thread? Just to prevent more discussion after the casting of the final vote?

(8) By Stephan Beal (stephan) on 2019-11-29 20:20:06 in reply to 7 [link] [source]

Back when we were discussing the creation of the forum feature, I asked for the same thing, but now having seen one-forum-per-repo in action, I wonder if this is a reactionary wish for a return to vBulletin/phpBB, Usenet, and FidoNet.

FWIW, i'm going to +1 that several times. +1. And again +1.

This forum software is "intended" to act as a forum for a single software project of the scale Fossil typically targets (normally a single library or app with, at most, a modest-sized development team and user base). It "could" arguably be split into "user support" and "developers" sub-forums, but, as Warren says, two instances solve that problem with little extra effort. For users, it's what amounts to two mailing list subscriptions, and for admins it's not all that much extra work (and requires zero extra implementation effort in the forum software).

Development of this forum software was a means to an end: getting off of the spam-infested mailing lists. It was not intended to be/become a full-time project on its own. (Richard may contradict/correct me on that, but i will be extremely surprised if he does!)

Not to mention that some of the features being requested, e.g. locked topics, cannot reasonably be implemented in a distributed system like this one, for the same reasons that file locking cannot be implemented 100% reliably (for a given definition of "100%," of course).

i, for one, hope that we can resist the urge to turn into a full-featured one-instance-hosts-100-subforums-and-thumbs-and-social-features piece of software. That way lies madness - there are already many full-featured, multi-tenant forum solutions out there.

IMO, YMMV, and all that.

Why? All that does is push the thread out of people's minds and then force those interested in voting to re-visit the thread later and re-read what's been said so they can cast their vote.

@Warren: i understood (perhaps incorrectly) the requested voting feature to be something more along the lines of BoardGameGeek's or mewe's: a user creates a poll with X possible answers, upon which users can vote, with an optional deadline, after which the poll is locked. After posting the poll, the list of answers/options is fixed - even the OP cannot edit them (to keep a malicious OP from swapping/changing answers around to misrepresent their desired outcome).

In any case, i consider it to be way out-of-scope for this software. (Though sometimes useful, i don't see the effort as being worth it for this project, frankly.)

(9) By Warren Young (wyetr) on 2019-11-29 20:34:13 in reply to 8 [link] [source]

On the last point, that is one of the things that SMAH was asking for, but the question you quoted is in answer to his request for the inverse situation: delay the start of voting until some later date. I'm asking why in a technical forum you wouldn't want to just up-vote all of the "good" posts as are currently present at the time you view them, rather than be forced to wait until November 6 or whatever to cast a single vote, as in a political election.

(15) By SMAH on 2019-12-04 14:19:00 in reply to 9 [link] [source]

I wasn't talking about upvoting posts, I was talking about attaching a voting form (stored in a new "voting form" table) to posts or threads. This is quite different to typical discussion forum upvoting. The purpose of the form would be to reach, confirm and record agreement in the database in discussions that would be conducted in the manner of a meeting ie. with a fixed point by which an action or motion must be taken. This would be more in line with a development or project workflow than a typical Internet discussion forum.

An example of this would be a monthly meeting where an agenda would be posted in the thread, a set of discussions would take place, the forum would be locked in say 3 weeks, the thread moderator (meeting chair) would post a minutes of meeting in a post (the thread moderator is allowed to do that), along with a voting form attached to the minutes of meeting post which asks for a vote on agreement of the minutes as a true record. Each member of the thread with voting rights (the attendees of the meeting) would vote to confirm agreement or not. If all agree, then the post remains locked. If anyone does not agree that the minutes are a true record, then the thread may be unlocked for further discussion as to what wasn't agreed and the revised minutes posted voted on again. Alternatively the disagreement may be discussed in the next monthly meeting and the voting form posted there.

As I said, this is not what most people think forum voting is all about. It is more of a development or project workflow process the results of which are embedded into the project database for posterity. I believe this is far more useful to a the development project process than the kind of populatity process upvoting we see on typical Internet discussion forums. It can also be used to get public opinion including anonymous opinion votes, with the advantage that the vote is counted in one place and accessible centrally on a per voting form basis and locked on conclusion, rather than having to tot up individual upvotes spread over a number of posts which are continually changing.

It requires per thread priviledges for moderators/moderator groups and voters/voter groups because different threads will be created/managed by different people, and voting may be limited to certain people.

(10) By anonymous on 2019-11-29 22:18:54 in reply to 8 [link] [source]

The poll could be the result of a CGI running under fossil.

The CGI has a "create poll" mode that takes a series of choices, start and end dates and a thread ID where it can post the result. It returns a "take vote" url that can be used in the thread to allow people to vote.

When the CGI is invoked after the end date and it sees that the poll results have not be posted, the script uses the (hypotherical) fossil command line to post the result to the thread.

Edits to the result (by the OP or other) would be tracked by fossil. If the OP is the admin, they can always deconstruct and reconstruct the fossil to change reality but ...

-- rouilj

(17.1) By SMAH on 2019-12-04 16:13:45 edited from 17.0 in reply to 7 [link] [source]

Back when we were discussing the creation of the forum feature, I asked for the same thing, but now having seen one-forum-per-repo in action, I wonder if this is a reactionary wish for a return to vBulletin/phpBB, Usenet, and FidoNet.

If you have multiple independent, non-overlapping topics of discussion, then do you not also probably need multiple Fossil repos to track the underlying artifacts under discussion?

The point here is that the Fossil forum is not there just to host the forum for the Fossil-scm site - Fossil is intended to host all sorts of development projects, and one size does not fit all.

It is not necessary to dictate what the end user has to adopt, and is very straightforward to allow multiple forums, and a single forum at the same time all according to the user's preferences. My concrete proposal for this is a fairly minor change that would be highly customisable and backward compatible with existing repositories. I am not suggesting a major change here. All I am suggesting is that the capability for customization is built in in a similar way to tickets. This can be done by allowing the addition of an extra field (say "topics") which can be optional to the thread table and allowing th1 scripting to produce different views (reports) of the forum thread table based on different selections based on this "topic" field, and at the same time you can display a one table for all view (your personal prefererence) by listing all threads.

There are big disadvantages in hosting threads in different repositories - having to set up files in different places, and losing the ability to search through every forum thread regardless of topic.

In what meaningful way is this feature not functionally equivalent to the following? $ fossil serve --repolist /path/to/forum/repos

Is this command in the documentation?

Which Fossil forum is bigger than this one? As one of its small number of moderators, I would not describe the duties of my job here as "excessive."

I'm presuming that you can point to a public Fossil forum, since in a private repo, what you need is not moderators but a culture of social norms that avoid the need for moderators. Put another way, moderation and the org chart are functionally overlapping concepts.

If you're asking instead for Fossil's user capability system to become flexible enough to model your employer's org chart, I don't see that happening.

To be fair, the purpose of the Fossil forum is not solely to host the Fossil-SCM forum - it is going to be used in lots of different teams and contexts. Hence I would consider flexibility to be paramount.

I think you have gotten hold of the wrong end of the stick regarding my comments on the forum feature. The advantage of Fossil over other scm systems is the fact that all the features are traceable and integrated into one database. There are many other ticket tracking systems and discussion forum systems on the market, most with better capability and more features. Where Fossil shines over the others is the tight integration of the ticket and wiki documentation into the project workflow. The one part of Fossil which is not tightly integrated into the development workflow is the forum feature. If one follows your views using separate repos to support separate forums to its logical conclusion, should probably be replaced with a separate better forum program. I am not of this view. My suggestions for the forum feature are intended to make it and its integration into the Fossil repository database more useful and more effective, and make a genuine case for using the Fossil forum instead of a separate forum program. The storing of voting forms, locking of threads, and groups of users associated with voting and posting on different threads, and saving these voting forms and threads in the database as a formal record of technical discussions is part of this integration with the development workflow. I believe there is a lot of utility in this.

Add the ability by the thread moderator or thread creator to set a list/group of users who are able to post on each thread, and by the moderator to set a list/group of users to whom the thread will be visible. This falls out of the prior item. If you have a need for private forums, you can get that today by having multiple repos, one per forum, each with a distinct set of users, each with different user capabilities, so that not all users can participate on all forums. Inversely, if you had this ability, how would you get it without greatly complicating the user capability system? Be concrete: propose a design.

This is just a suggestion for the expansion of the user/group capabilities already present in Fossil for the forum feature. It would not be appropriate to put different code into different repositories just to control access, for the same reason that user capabilities are currently built into Fossil. In a development team, you will have certain people responsible for certain actions, although all may need access to the code. I am suggesting that the thread user/group capability matrix be stored in the thread and be per thread, and that the thread moderator capability also be per thread (being assigned to both the thread creator or the forum moderator). The per thread capability relates mainly to thread locking,unlocking,voting, and posting capabilities - it is not necessarily intended to be private. This is not particularly complicated. Fossil is also not always going to be used to develop open source software, so there may be a need to use the master/child repository feature to keep some things private, in which case the private discussion forum can be hosted in the child repository.

(18) By Stephan Beal (stephan) on 2019-12-04 17:16:03 in reply to 17.1 [link] [source]

I am suggesting that the thread user/group capability matrix be stored in the thread and be per thread, and that the thread moderator capability also be per thread (being assigned to both the thread creator or the forum moderator). The per thread capability relates mainly to thread locking,unlocking,voting, and posting capabilities - it is not necessarily intended to be private.

One critical detail which i get the impression you're disregarding: fossil's forum model is, like its SCM features, distributed. That means, for example, that thread locking cannot be reasonably implemented for essentially the same reasons file locking cannot (for a reasonable definition of "reasonably"). Anyone can clone it, make changes locally and, with the right permissions, push those to any other copy of the repo. If a central copy chooses to reject (e.g., because of a thread lock or expiry timer) what's being pushed, it effectively forks that forum, leaving the clone in a state which can never be reconciled with the origin copy.

(19) By SMAH on 2019-12-04 21:53:35 in reply to 18 [link] [source]

But would you ever want to push the clone of a forum up to the server from which you cloned/pulled the repository? That option should be disallowed.

You would want to pull FILES down from the master repo, make changes to them locally, do some builds, test them and once the tests pass, push them up to the master. This makes perfect sense because to do the build requires fast access to a large number of files so you need these locally, and you would want to test them before pushing them back up, so a distributed SCM makes sense for files.

For forums this doesn't make sense. You are making posts one at a time, and you want everybody else on the forum to see the latest posts immediately. You certainly do not want to make the posts locally and then push them to the server in one weeks time. You want everybody on the forum to see everybody's posts immediately. You may want to pull the forum down to the local repo to preserve a backup for reasons of legal record, Quality Assurance, or data redundancy. The forum should be centralized and server based.

The way you would want this to work is that when you clone a repository with a forum, the forum would be copied down locally, but you would not be able to make posts to the local repository. Instead when you cloned the master repository, an HTML link to the master repository's forum would be copied down, and when you try to post to the repository, you would connect to master repository and the post would be made to the master repository forum, and immediately after, a pull would be done to pull down the changes to the master repository forum down to the local repository for the forum only. If someone cloned from your local repository, then the forum link would be copied down from your local repository, so it would point to the same master repository, so everybody should be posting to the same master repository. However there needs to be a mechanism for pulling the forum from the master repository to the local repository, and then from the local repository to the clone of the local repository.

There may be a need for a local forum for say internal/private discussions between the local build and testing teams on test builds. For this you would set up another forum on the local repository (which would be private to the local repository and not be pulled from the master repository). The forum link for this including the clones of the local repository would refer to the server on the local repository. This is another reason for supporting multiple named forums.

(20) By Stephan Beal (stephan) on 2019-12-04 22:26:03 in reply to 19 [link] [source]

It's not a question of wanting or not wanting this behavior, it's a simple fact of the distributed model that there is, at the technical level, no master, no one single source of Truth. In practice we tend to think of fossil-scm.org as the central fossil forum server, but that's a convenient illusion inherited from more traditional computing paradigms (where it is truth, not illusion). DVCSes and, by extension, this forum software, simply don't operate in a way which involves a primary/master node. On the contrary, it's an operating model they explicitly disavow.

(24) By SMAH on 2019-12-05 10:03:26 in reply to 20 [link] [source]

I am sorry - what you are saying makes no sense whatsoever.

Firstly, distributed SCM software needs to be syncronised from time to time with a master repo somewhere to merge changes, otherwise the code in each local clone will diverge from every other. That is a basic requirement of distributed SCM. For a centralised SCM you don't need to syncronize because everybody works off the same master repository. If what you claim is true, then what do you think the Fossil pull and push commands are for?

Secondly, discussion forums should work off a centralised server, and cannot sensibly for the reasons I have given, work in the way you claim (ie. make the posts locally to the local cloned copy, and then push up to the master repo for others to sync with your posts at a later date to read them). In fact Fossil doesn't seem to work the way you suggest way either - if you read the documentation on the Fossil "push" and "pull" commands, they explicitly state:

Push:
"Push all sharable changes from the local repository to a remote repository. Sharable changes include public check-ins, and wiki, ticket, and tech-note edits."

Pull:
"Pull all sharable changes from a remote repository into the local repository. Sharable changes include public check-ins, and wiki, ticket, and tech-note edits."

...no mention of forums - so Fossil forums do not appear currently to be distributed as you have claimed - which kind of makes sense.

Having said that, it would as I have suggested, make sense to allow one-way pulling down (but not pushing up), and viewing of a local copy of the discussion forum from the master repo it was cloned from for record purposes and for tracing development history, especially if the forums are to contain important records pertaining to development interaction like voting records, and things agreed in the forums. It would make sense in view of Fossil's philosophy of capturing all key data pertaining to development, to pull down and keep a local copy of the master discussion forum synced locally while directing the links to the forum to the centralised forum server in the master repository. This local record could also be important for contractual, legal, and quality reasons if the forum is to be used as an alternative to email for example for key communications or agreements between collaborators. As I said before, you would probably also want support for a private forum for your local team discussions served off your local clone, which is stored on the local repo but not synced with the master repo (hence multiple forum support to keep them separate).

(25) By Stephan Beal (stephan) on 2019-12-05 10:44:58 in reply to 24 [link] [source]

Firstly, distributed SCM software needs to be syncronised from time to time with a master repo somewhere to merge changes,

That is a an overwhelmingly common usage pattern, not a requirement of the distributed data model or software. The model very specifically does not require syncing with a master/central copy.

otherwise the code in each local clone will diverge from every other.

Which is a perfectly valid way to use a distributed system. Very probably exceptionally rare, but completely valid.

Secondly, discussion forums should work off a centralised server, and cannot sensibly for the reasons I have given, work in the way you claim (ie. make the posts locally to the local cloned copy, and then push up to the master repo for others to sync with your posts at a later date to read them).

"Should," perhaps, but the data model and implementation both allow the forum to be decentralized. Restricting that potential/possible workflow via project policy would require an intentional hobbling of the software to not make use of its full feature set.

i'm not saying that project policy cannot work that way, but the software, as designed, does not have to, and there's nothing in the world, other than agreed-upon/handshake project-level policy, keeping the forum software from being operated this way.

i agree entirely, 100%, that a central server is the most common, intuitive, and natural way for forums to operate. That's not how this software is modeled, though. It's a distributed system, and being distributed inherently brings with it certain features and limitations. One of those limitations is the effective inability to reliably lock records after their creation (forum posts, threads, and polls, in this case).

That said...

When using a centralized server and pull-only cloning (if cloning is allowed at all), locking could reasonably be implemented and would work... until someone gets permission to push from a clone, at which point the locking logic breaks down and a locked post on the central copy could keep a user from ever being able to push their clone (or possibly even pull if they've locally locked threads for which new replies are being pulled). Pushed locks, as well, could potentially disrupt the central forum by locking threads which were not previously locked on the central copy. The only line of defense against that, in this software, would be intentionally disabling/not using the distributed features (which includes the ability to read and reply to posts when completely offline, e.g., from an airplane).

i'm not at all discounting that your locking/polling suggestions are useful in a general-purpose forum, but am convinced that they're not a realistic fit for forum based on a distributed data model. (Sidebar: fossil's forum may well be the only such software?)

(26) By Stephan Beal (stephan) on 2019-12-05 10:51:59 in reply to 24 [link] [source]

PS:

Push:

"Push all sharable changes from the local repository to a remote repository. Sharable changes include public check-ins, and wiki, ticket, and tech-note edits."

Pull:

"Pull all sharable changes from a remote repository into the local repository. Sharable changes include public check-ins, and wiki, ticket, and tech-note edits."

...no mention of forums - so Fossil forums do not appear currently to be distributed as you have claimed - which kind of makes sense.

Because the documentation for pull and push largely predates the forum software by a decade and hasn't been updated (i'll remedy that in a moment). Fossil's forum data model and implementation is the exact same as the core SCM's. (Forum posts are internally a small extension of wiki pages.)

(11) By sean (jungleboogie) on 2019-11-30 22:49:59 in reply to 1 [link] [source]

Hi drh,

I agree with your assessment of how well the mailing list is working out. Thanks for creating it and having the desire to make it better.

My suggestion/request may already be considered and covered by your thoughts, but I'll spell it out for clarity.

The long lines of text in the forum makes for lots of horizontal reading across many long lines. I know this can be adjusted by making the browser smaller (I don't run Firefox full screen), but then the layout on other sites may not be ideal. Friend of the forum, Andy Bradford posts in plain text and likely uses something outside the forum to control his margins.

Small example here: https://fossil-scm.org/forum/forumpost/68d2d04b93

So my request is to make margins (or whatever it is) more in the 80 column rule that the code follows.

(12) By anonymous on 2019-12-03 23:43:13 in reply to 1 [link] [source]

(4) Built-in Support For PIC and EQN

I don't know much about PIC, but I would suggest that support for ASCIImath would be better then EQN. thanks to MathJax, ASCIImath is a very popular markup for equations and formulas on the web.

Your idea for putting PIC code in a ~~~ block would work for ASCIImath. Or, MathJax looks for $$ ... $$ or \[ ... \] as delimiters.

(13) By Stephan Beal (stephan) on 2019-12-04 10:31:28 in reply to 12 [link] [source]

A cursory glance suggests that ASCIImath only supports math formulas, though, whereas the proposed feature for PIC/EQN/whatever is intended to support drawing of fossil inheritance diagrams and such.

Related: it seems that there at at least 2 JS libraries implementing the DOT language:

https://en.wikipedia.org/wiki/DOT_(graph_description_language)#JavaScript

(That said, i believe that Richard's desire is for a server-side implementation.)

(14) By Richard Hipp (drh) on 2019-12-04 12:34:35 in reply to 13 [link] [source]

I don't have strong feelings about client-side vs. server-side. Features I am looking for:

  • A mature and stable language that is either well-supported or which is simple enough for us to support ourselves. I want documents written today using the drawing- or equation-language to be viewable in the year 2050 with as little maintenance as possible in the interim.

  • The drawing language needs to be adequate for generic diagrams - not just graphs.

  • Has an open-source license compatible with the 2-clause BSD that Fossil uses.

  • Does not impose excessive tertiary dependencies

  • The language should be well-documented and easy enough to learn that people are likely to learn it and use it.

  • Does not use excessive bandwidth nor use excessive CPU on either client or server.

  • Can be integrated into Fossil without having to do major refactoring of the code, source tree layout, or build infrastructure.

I glanced briefly at MathJax last night. It seems like it might fit the bill as an equation-language. I was going to do an experimental integration on a branch, when I get a chance.

(21) By SMAH on 2019-12-04 23:53:19 in reply to 14 [link] [source]

MATHS

MathJax is a cut down version of the Latex typesetting language that supports mathematical equations implemented in Javascript in browsers with HTML5 MathML support. https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference

MathJax is also supported in many editors and renderers:

DocuWiki: https://www.dokuwiki.org/plugin:mathjax

Markdown: https://hiltmon.com/blog/2017/01/28/mathjax-in-markdown/, https://stackedit.io/ https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html

HTML: https://www.mathjax.org/#gettingstarted.

DIAGRAMS

SGV is supported as it is part of HTML5

Some Markdown renderers can display SVG eg: Jupyter Notebook https://stackoverflow.com/questions/44139441/inlining-svg-image-in-a-markdown-cell GitHub Markdown: https://stackoverflow.com/questions/13808020/include-an-svg-hosted-on-github-in-markdown DocuWiki support: https://www.dokuwiki.org/plugin:svg

UML is another format for diagrams. It has HTML javascript libraries, and the stackedit.io markdown editor supports it, but I am not aware that it is widely used.

The problem is to find a Wiki or Markdown format that can be considered standardised and enduring. The MathJax, and equation subset of latex and is reasonably human readable, so it is probably worth including with some sort of standard enduring Wiki/Markdown format, ideally with a richtext embedded editor. If the format is too complex or too human unreadable, then I think it should be left out of the text input, as Fossil is not a content management platform. Instead it would be more appropriate to include those richtext formats as file attachments or artifact ID references with MIMEtypes that would render via external programs when opened.

(30) By anonymous on 2019-12-06 20:03:30 in reply to 21 [link] [source]

Some Markdown renderers can display SVG

Fossil's markdown implementation (and its wiki implementation) allow some HTML. The HTML sanitizer would need to be enhanced to allow SVG.

The problem is to find a Wiki or Markdown format that can be considered standardised and enduring.

Fossil's wiki format is a subset of wiki features from several different wiki formats. It's simple and readable, so could be considered enduring in the sense that a reasonable rendering engine could be devised just by looking at a few documents that use Fossil Wiki format.

Fossil's markdown format is close to being a subset of CommonMark.

(34) By SMAH on 2019-12-08 02:09:16 in reply to 30 [link] [source]

Some Javascript libraries for generating svg diagrams. https://modeling-languages.com/javascript-drawing-libraries-diagrams/

(39) By anonymous on 2019-12-08 23:34:41 in reply to 34 [link] [source]

Some Javascript libraries for generating svg diagrams. https://modeling-languages.com/javascript-drawing-libraries-diagrams/

Most of those appear to depend on node.js or similar.

mxGraph claims to have no dependencies. But, other than being an underlying library for draw.io, it's not clear whether it can be used directly to render diagrams from a simple textural description or would require a parser, etc that call mxGraph.

(40) By SMAH on 2019-12-16 03:08:45 in reply to 14 [link] [source]

Another thought about the forum post text format: If you want a universal long enduring format, how about html email format with file attachments stored as Base64 encoding and fossil artifact attachments stored as the URL used to view the artifact? To maintain compatibility, this requires existing legagy Fossil formats (Markdown, Wiki, Text), to be converted to email format on import - which shouldn't be too difficult (Markdown MathJax equations can be converted to MathML which is part of HTML5 standard), plus there are a lot of viewers and composers that can handle this format. This format is standard and will be enduring. It also has some other big advantages, namely:

  1. Conversion to and from email is easy which means legacy mailing lists can be imported easily, and the project can be continued on a Fossil forum.
  2. Users who want to carry on working in mail list format can carry on doing so - they opt for notification on a forum thread or topic, and they get an email which has a return address that points to a forum topic (eg. "Post/ThreadID@forum_topic.repoURL" with a subject that corresponds to the thread title, and the poster's Fossil username appearing as the sender's name, and the cc list appearing as the mailing list users and groups (this format would be used for all posts). When the user replies by email, the email is inserted into the forum under the correct thread/post. The forum_topic here would be an extra field in the fossil forum thread table which would allow selection based on the forum topic if required to allow simulation of multiple forums id required.
  3. Users can put the entire project workflow into Fossil and exclude email completely if required. For example, from the Fossil post composer, it would be possible to include outside parties who are not users of or subscribed to the forum, by including an email address as an addressee. By default the cc list should instead probably be a bcc list, and this feature should be restricted to key group - for example members of the development team and probably only allow whitelisted email addresses (to avoid use of Fossil for spamming). Otherwise the format is similar to 2) above. When the external user replies, the reply is posted in the correct place in the forum. This would be useful for including external patry communication which may be important for contractual or legal reasons in the repository - for example the email could ask for clarification on license terms for a software library from another project, and the response could be recorded in the conversation thread. Another example would be used to record the sending out of user terms and conditions to participants in a development project to their real email addresses, and record their acceptance in their email reply for contractual and legal reasons.
  4. Users can forward external emails from their email client to a particular post ID in the format in 2) above in order record this in a post. You would create a post explaining what the email you are going to attach is about, and then use a short ID number and one time password hidden from others to forward the email to. It would appear as a reply to the post, but in a another colour to signify that it is an attachment of an email rather than an independant post. This is useful for retrospectively posting external emails/mailing list emails that have been received/sent in the past (for example the details in the email may have had to be kept confidential until signing of the contract for legal, contractual, or team working reasons).
  5. It could allow Fossil to be used for archiving up other project email on an email server or email client (for example held in Unix maildir or mbox format) at the end of the project or at project stages. This would require a Fossil import program which would create a new named forum topic, and import a mailbox file or directory into it as conversation threads based on mailbox directory structure and posts based on subject and replies. This would be used to archive in stages important external project emails which have not been deleted or have been moved into a shared email folder in so as to include capture of project emails as an immutable record.
  6. It could allow mirroring of posts by forwarding/relaying emails of threads/posts to an externally hosted third party email archiving service for legal/contractual reasons or as a backup. This is sometimes a requirement when different contracting parties collaborate - for example in most contracts the minimum limitation period for retaining emails which have contractual importance is 12 years, and for personal injury it is 6 years from the time that the injury should reasonably have become apparent (maybe important for safety systems).

You are probably wondering why I have suggested enhancements like email, mailing list, multiple topic forum, and collaborative workflow enhancements like voting forms and more granular access and permissions control for topics. The reason is because Fossil is really neat and unique in that it tries to capture everything in a project and saves it in one immutable blockchain protected database file which can easily be distributed as a file. The big gap in Fossil though is email. Email has contractual importance and needs to be archived for a long period in time 12 years or more typically, and possibly deleted promptly after this in order to avoid liability for disclosure after this period. The problem with emails is that they are horrendous to archive because all projects end up in a user's mailbox, the mailbox is a list rather than a tree structure which makes it difficult to see conversations, and reliance has to be placed on user discipling to title the email subject to reflect the project and conversation thread in order to allow different projects to be archived separately or allow conversation threads to be followed. In practice people are lazy, and this never happens. Because of this medium to large companies buy email archiving services that pay for all company wide emails to be archived over the network/Internet, and allow searching capability in case emails have to be retrieved for litigation later. These are then usually deleted after a period of time (which is often a problem because the retention period may need to be different for different projects). Mailing Lists solve this partially by enforcing some rigour in this, but are in my opinion still problematic because they clutter up your inbox - forums are a better option to emails in my opinion because they keep emails out of your inbox.
mailing lists vs forums
The ideal solution would be if use of email can be eliminated entirely from project workflow (except for notification or as a back-up) by expanding the capabilities of forums to enforce capture of all project workflows that may use emails. This is what my suggestions aim at doing. The ability for Fossil forums to replace all mailing list functionality, have granular permissions and private forums, and their ability to capture contractually important emails that do not go through the forum aim at doing this. The voting form suggestion is aimed at replacing another workflow that usually goes through emails - for example confirmation of agreement by all on minutes of meetings, accepting actions or tasks allocated, confirmation of agreement to terms and conditions etc. I think the ability to record this and enforce capture of all project workflow records including emails and alternatives to email, with complete separation of project archives from each other, and save this in an immutable file which you yourself can keep archived or distribute as you see fit is a huge advantage over what happens now, is a huge selling point for using Fossil for development projects, especially smaller ones. As I said, it is what I think is required to complete Fossil's intention to capture the whole development process.

(41) By ss48 on 2019-12-16 21:47:05 in reply to 14 [link] [source]

There is also another math layout engine for equations called KaTeX which also seems to be pretty popular (KaTeX) and might be worth investigating. It's MIT Licensed.

(42) By SMAH on 2019-12-16 23:45:52 in reply to 41 [link] [source]

Fossil html file artifacts with MathML render math equations OK 

Try:

<!DOCTYPE html>
<html>
<head>
<title>MathJax MathML Test Page</title>
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script type="text/javascript" id="MathJax-script" async
  src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/mml-chtml.js">
</script>
</head>
<body>

<p>
When
<math xmlns="http://www.w3.org/1998/Math/MathML">
  <mi>a</mi><mo>&#x2260;</mo><mn>0</mn>
</math>,
there are two solutions to
<math xmlns="http://www.w3.org/1998/Math/MathML">
  <mi>a</mi><msup><mi>x</mi><mn>2</mn></msup>
  <mo>+</mo> <mi>b</mi><mi>x</mi>
  <mo>+</mo> <mi>c</mi> <mo>=</mo> <mn>0</mn>
</math>
and they are
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
  <mi>x</mi> <mo>=</mo>
  <mrow>
    <mfrac>
      <mrow>
        <mo>&#x2212;</mo>
        <mi>b</mi>
        <mo>&#x00B1;</mo>
        <msqrt>
          <msup><mi>b</mi><mn>2</mn></msup>
          <mo>&#x2212;</mo>
          <mn>4</mn><mi>a</mi><mi>c</mi>
        </msqrt>
      </mrow>
      <mrow>
        <mn>2</mn><mi>a</mi>
      </mrow>
    </mfrac>
  </mrow>
  <mtext>.</mtext>
</math>
</p>

</body>
</html>

Fossil html file artifacts with MathJax do not render properly however. 
Try:

<!DOCTYPE html>
<html>
<head>
<title>MathJax TeX Test Page</title>
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script type="text/javascript" id="MathJax-script" async
  src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js">
</script>
</head>
<body>
When \(a \ne 0\), there are two solutions to \(ax^2 + bx + c = 0\) and they are
$$x = {-b \pm \sqrt{b^2-4ac} \over 2a}.$$
</body>
</html>


I believe MathJax requires a Javascript library which the Fossil renderer doesn't use even though the browser I am viewing it with renders both MathJax and MathML. MathJax is supported by most web browsers and Markdown renderers though and is more popular and more human readable, so it should be used as well. It is aslo important because Jupyter Notebooks outputted as HTML do not render properly in Fossil file artifacts.

Neither format works properly in wikis or markdown

(43) By anonymous on 2019-12-17 06:11:11 in reply to 42 [link] [source]

I believe MathJax requires a Javascript library which the Fossil renderer doesn't use

MathJax is a Javascript library that runs in the browser.

Some browsers are able to directly parse and render MathML. Maybe this is what happened.

Can you be sure the MathJax library was actually used? By default, Fossil tells browsers to not load third party Javascript (or CSS or several other things).

(44) By SMAH on 2019-12-17 13:17:08 in reply to 43 [link] [source]

Well that may explain it. It seems MathML is part of the official HTML5 spec, MathJax seems to be a client side Javascript library. If Fossil is telling the browser not to load the MathJax Javascript libraries, then that is probably why MathJax is not working in Fossil.

Comparing MathJax and MathML

MathML is part of HTML5

Can install MathJax libs on your own server (or use a Content Distribution Network service)

MathJax is useful for Markdown, and some HTML documents eg. Jupyter Notebooks, as well as for compatibility. Any idea how to enable it to be used from a CDN service in Fossil?

(45) By Joel Dueck (joeld) on 2019-12-17 13:45:04 in reply to 44 [link] [source]

I’m not sure if this is the best place for this reply. But whenever I needed additional capabilities requiring Javascript, I just downloaded the standalone .js file and added it to the Fossil repo as an unversioned file, then edited the theme’s header to load it from there. This avoids third-party restrictions (although I believe these too can be removed if desired, not a good idea though) Is there some reason people can’t do this?

(46) By SMAH on 2019-12-17 22:40:03 in reply to 45 [link] [source]

The design decision to not allow external Javascript libraries in Fossil kind of makes sense from a security point of view given that html files in Fossil can't be trusted due to anybody being able to upload html files with malicious Javascript into a Wiki or submit it as a file into the repository. The lack of Javascript permitted in html emails is also probably there for the same reason. Anyway, this is something which needs to be considered very carefully. https://superuser.com/questions/536569/math-formulas-in-emails

The solution would therefore seem to be to dissallow external Javascript libraries and include a local copy in the Fossil distribution which Javascript in Fossil would reference. The problem with this is that the local copy of the MathJax libs will have to be updated from time to time, which could be an issue. The other alternative would be to use MathML only and wait until the HTML standard includes MathJax as part of the core standard, and use an embedded MathML formula composer in Fossil for forum posts in HTML email format, and wikis, and to exclude MathJax equations from Markdowns and Wikis
https://en.wikipedia.org/wiki/Formula_editor

MathML in HTML5: Finally cross-browser HTML+MathML!

Web browsers used on personal computers either support MathML directly or they can rely on MathJax, an open source MathML display engine. MathJax uses native MathML rendering in browsers that have it and JavaScript-generated HTML and CSS in browser that don't such as those in smartphones, tablets, and ebook readers.

I presume what is described here parallels what you have done. I think that may be the way to go for rendering views of Fossil HTML artifacts with equations (eg. Jupyter Notebook generated documents), or just include a warning that the view contains external Javascript that is not trusted and may be downloaded viewed more correctly with an external browser.

Installing your own copy of MathJax

Installing Your Own Copy of MathJax
We recommend using a CDN service if you can, but you can also install MathJax on your own server, or locally on your own hard disk. To do so you will need to do the following things:

  1. Obtain a copy of MathJax and make it available on your server or hard disk.
  2. Configure MathJax to suit the needs of your site.
  3. Link MathJax into the web pages that are to include mathematics.
  4. Put mathematics into your web pages so that MathJax can display it.

Obtaining and Installing MathJax

  • The easiest way to set up MathJax is to obtain the v2.0 archive from the MathJax download page (you should obtain a file named something like mathjax-MathJax-v2.0-X-XXXXXXXX.zip where the X’s are random looking numbers and letters). This archive includes both the MathJax code and the MathJax webfonts, so it is the only file you need. Note that this is different from v1.0 and earlier releases, which had the fonts separate from the rest of the code.

  • Unpack the archive and place the resulting MathJax folder onto your web server at a convenient location where you can include it into your web pages. For example, making MathJax a top-level directory on your server would be one natural way to do this. That would let you refer to the main MathJax file via the URL /MathJax/MathJax.js from within any page on your server.

  • Note: While this is the easiest way to set up MathJax initially, there is a better way to do it if you want to be able to keep your copy of MathJax up-to-date. That uses the Git version control system, and is described in the Installing MathJax document. If you prefer using Subversion, you can also use that to get a copy of MathJax (see Installing MathJax via SVN).

  • Once you have MathJax set up on your server, you can test it using the files in the MathJax/test directory. If you are putting MathJax on a server, load them in your browser using their web addresses rather than opening them locally (i.e., use an http:// URL rather than a file:// URL). When you view the index.html file, after a few moments you should see a message indicating that MathJax appears to be working. If not, check that the files have been transferred to the server completely and that the permissions allow the server to access the files and folders that are part of the MathJax directory. (Be sure to verify the MathJax folder’s permissions as well.) Check the server log files for any errors that pertain to the MathJax installation; this may help locate problems in the permission or locations of files.

(48) By eric b. (ebcfr) on 2020-01-10 14:27:14 in reply to 14 [link] [source]

Hello,

This page might be of interest for equation rendering
https://developer.mozilla.org/en/docs/Web/MathML/Authoring

I played a little with LaTeXMathML (https://github.com/asciimath/asciimathml) which translates LaTeX equations to MathML.

Equations are given in the form $eq$

I patched markdown.c and markdown_html.c to get raw equation sent unchanged into the html output, and tested it with a markdown file after putting all the LaTeXMathML.js (minus the two lines with <script> int the beginning comment) in the js.txt of the skin (it's about 50k, probably much less than you will get with MathJax). When '$' is used in another place than equations, it has to be escaped with \\$. '$' can be escaped as well in equations.

Here are the patches. They are against fossil-2.10. You can make what you want of it, Same for the example, which is a mere translate to markdown of the test page 'test/lmtest.html' from the github repository.

I also played a little with the ASCIIMathML syntax although less widely used than the LaTeX one. I had to change the line 54 of ASCIIMathML.js
var AMdelimiter1 = "`", AMescape1 = "\\\\`"; --> var AMdelimiter1 = "$", AMescape1 = "\\\\$";
But I'm then unable to get escaped '$' elsewhere in the markdown file.

Eric

markdown.patch
--------------
--- markdown.c	2019-10-04 23:41:13.000000000 +0200
+++ math/markdown.c	2020-01-09 18:11:03.951960818 +0100
@@ -73,6 +73,7 @@
   int (*double_emphasis)(struct Blob *ob, struct Blob *text,
             char c, void *opaque);
   int (*emphasis)(struct Blob *ob, struct Blob *text, char c,void*opaque);
+  int (*equation)(struct Blob *ob, struct Blob *text, char c,void*opaque);
   int (*image)(struct Blob *ob, struct Blob *link, struct Blob *title,
             struct Blob *alt, void *opaque);
   int (*linebreak)(struct Blob *ob, void *opaque);
@@ -718,6 +719,52 @@
 }
 
 
+/* char_equation -- '`$' parsing a code span (assuming codespan != 0) */
+static size_t char_equation(
+  struct Blob *ob,
+  struct render *rndr,
+  char *data,
+  size_t offset,
+  size_t size
+){
+  size_t i;
+  char c = data[0];
+
+  /* whitespace cannot follow an opening equation */
+  if( data[1]==' '
+   || data[1]=='\t'
+   || data[1]=='\n'
+   || data[1]==c
+  ){
+    return 0;
+  }
+  
+  /* finding the next delimiter */
+  i = 1;
+  while( i<size ){
+    while( i<size && data[i]!=c){ i++; }
+    if( i>=size ) return 0;
+
+    /* not counting escaped chars */
+    if( i && data[i-1]=='\\' ){
+      i++;
+      continue;
+    } else if( data[i-1]==' ' 
+            || data[i-1]=='\t'
+            || data[i-1]=='\n'
+    ) {  /* no whitespace before the closing equaation symbol */
+      return 0;
+    } else {
+      break;
+    }
+  }
+
+  struct Blob work = BLOB_INITIALIZER;
+  blob_init(&work, data, i);
+  return rndr->make.equation(ob, &work, i, rndr->make.opaque) ? i : 0;
+}
+
+
 /* char_linebreak -- '\n' preceded by two spaces (assuming linebreak != 0) */
 static size_t char_linebreak(
   struct Blob *ob,
@@ -2182,6 +2229,7 @@
       rndr.active_char[(unsigned char)rndr.make.emph_chars[i]] = char_emphasis;
     }
   }
+  if( rndr.make.equation ) rndr.active_char['$'] = char_equation;
   if( rndr.make.codespan ) rndr.active_char['`'] = char_codespan;
   if( rndr.make.linebreak ) rndr.active_char['\n'] = char_linebreak;
   if( rndr.make.image || rndr.make.link ) rndr.active_char['['] = char_link;


markdown_html.patch
-------------------
--- markdown_html.c	2019-10-04 23:41:13.000000000 +0200
+++ math/markdown_html.c	2020-01-09 18:13:39.499335077 +0100
@@ -397,6 +397,16 @@
   return 1;
 }
 
+static int html_equation(
+  struct Blob *ob,
+  struct Blob *text,
+  char c,
+  void *opaque
+){
+  BLOB_APPEND_BLOB(ob, text);
+  return 1;
+}
+
 static int html_image(
   struct Blob *ob,
   struct Blob *link,
@@ -501,6 +511,7 @@
     html_code_span,
     html_double_emphasis,
     html_emphasis,
+    html_equation,
     html_image,
     html_line_break,
     html_link,


equations.md
------------
LaTeXMathML examples
====================

A page test for [LaTeXMathML.js](https://github.com/asciimath/asciimathml).

An inline equation with escaped \\$ characters: the input

    \$\\$\alpha + \\\$\beta = \\$(\alpha + \beta)\$

produces $\$\alpha + \$\beta = \$(\alpha + \beta)$. The input

    \$\lim_{x\to\infty} f(x) = k \choose r + \frac ab \sum_{n=1}^\infty a_n +
    \displaystyle{ \left\{ \frac{1}{13} \sum_{n=1}^\infty b_n \right\} }\$ 

produces $\lim_{x\to\infty} f(x) = k \choose r + \frac ab \sum_{n=1}^\infty a_n +
    \displaystyle{ \left\{ \frac{1}{13} \sum_{n=1}^\infty b_n \right\} }$. Changing \displaystyle to
\textstyle and `\$...\$` to `\$\displaystyle{ ... }\$` gives

$\displaystyle{ \lim_{x\to\infty} f(x) = k \choose r + \frac ab \sum_{n=1}^\infty a_n +
    \left\{ \frac{1}{13} \sum_{n=1}^\infty b_n \right\} }$ 

The input

    \$\begin{eqnarray} x & = & \frac{-7 \pm \sqrt{49 - 24}}{6} \\
    & = & -2 \textrm{ or } -\frac13. \end{eqnarray}\$

produces

$\begin{eqnarray} x & = & \frac{-7 \pm \sqrt{49 - 24}}{6} \\
    & = & -2 \textrm{ or } -\frac13. \end{eqnarray}$

(LaTeXMathML tries to reduce the column spacing in eqnarrays, but some browsers do not respond, and others over-react.) The input

    \$\displaystyle{ f(x) := \left\{\begin{array}{l l}
    x^2 \sin \frac1x & \textrm{if } x \ne 0, \\
    0 & \textrm{if } x = 0 .
    \end{array}\right.}\$

produces $\displaystyle{ f(x) := \left\{\begin{array}{l l}
    x^2 \sin \frac1x & \textrm{if } x \ne 0, \\
    0 & \textrm{if } x = 0 .
    \end{array}\right.}$

The input

    \$\displaystyle{ A = \left[\begin{array}{c c c}
              		   1-x & 0   & 0   \\
              		   0   & 1-x & 0   \\
              		   0   & 0   & 1-x \end{array}\right]\$

produces $\displaystyle{ A = \left[\begin{array}{c c c}
    1-x & 0   & 0   \\
    0   & 1-x & 0   \\
    0   & 0   & 1-x \end{array}\right]$

Large summation signs, etc., without limits can sometimes get too big. For example, the input

    \$\sum a_i + \sum_{i=0}^\infty b_i\$ 

produces $\sum a_i + \sum_{i=0}^\infty b_i$. In some browsers, the presence of the second summation sign, with limits, makes the first one too big. Enclosing the first one in braces, as in \{\\sum a\_i\}, produces ${\sum a_i} + \sum_{i=0}^\infty b_i$, which is arguably better. (This is a bug in LaTeXMathML, but one that would be difficult to correct.)

(31) By anonymous on 2019-12-06 20:18:58 in reply to 13 [link] [source]

A cursory glance suggests that ASCIImath only supports math formulas, though, whereas the proposed feature for PIC/EQN/whatever is intended to support drawing of fossil inheritance diagrams and such.

I wasn't suggesting not using PIC. I don't know what might be a better choice, if any.[1]

I was only suggesting the ASCIImath might be a better choice than EQN because ASCIImath seems to be a defacto standard for formulas and equations on the web.

[1] I know of simpler graphing/charting languages like dot and MSC, but those are for specific types of graphs/charts (dot for directed graphs and MSC for message sequence charts).

(32) By anonymous on 2019-12-07 00:35:26 in reply to 31 [link] [source]

I wasn't suggesting not using PIC. I don't know what might be a better choice, if any.

I hope there is a better choice.

PIC is very awkward to code in. For example, in https://ece.uwaterloo.ca/~aplevich/dpic/dpicdoc.pdf

> The construction
>
>     if condition then {if-true}
>     else {if-false}
>
> produces an error with all pic interpreters. To avoid this error, write
>
>     if condition then {if-true} \\
>     else {if-false}

Contrast that with

>     for d = 0 to 360 by 2 do {
>         line from (0,0) to (cos(d*dtor),sin(d*dtor))
>     }

Not consistent.

(47) By Stephen Weigand (weigand) on 2019-12-18 03:05:23 in reply to 32 [link] [source]

This is not a proposal but in case anyone finds this approach interesting, the Emacs Org mode documentation introduced me to ditaa which makes an image out of an ASCII diagram. Here's their example diagram:

+--------+   +-------+    +-------+
|        | --+ ditaa +--> |       |
|  Text  |   +-------+    |diagram|
|Document|   |!magic!|    |       |
|     {d}|   |       |    |       |
+---+----+   +-------+    +-------+
    :                         ^
    |       Lots of work      |
    +-------------------------+

It relies on the Java runtime so I assume not helpful for Fossil but the ideas may be interesting and the readable plain text format is consistent with the spirit of Markdown.

On the other hand, I would be happy to learn PIC and super happy to use EQN.

-- Stephen

(16) By SMAH on 2019-12-04 14:21:50 in reply to 12 [link] [source]

Adding latex equations as some Markdown formats are able to support (notably Jupyter Notebooks) would be a good idea, along with MathJax for the Wiki/HTML format.

(22.1) By Stephan Beal (stephan) on 2019-12-05 06:29:53 edited from 22.0 in reply to 1 [link] [source]

(2) Delta Compression Of Edited Forum Posts

Fossil is not currently using delta compression to minimize the size of forum posts that have been edited. It should.

Is it correct to say that this implementation simply requires doing the equivalent of the following:

  1. get the previous version's rid.
  2. content_put() the new version, thereby getting its newRid.
  3. call content_deltify(rid, &newRid, 1, 0), thereby deltifying the previous version (as opposed to the new one).

:-?

i'm still digging through forum.c to find where this is done, but all uses of content_deltify() seem to be that straightforward.

Edit/update: forum posts are saved by passing them from forum_post() to wiki_put(), which calls content_deltify() if a parent rid is passed to it and the post does not require moderation. We have the parent version's ID in forum_post()'s iEdit parameter, so this would seem to be a simple case of passing iEdit on as wiki_put()'s 2nd argument. Wish me luck...

(23) By Stephan Beal (stephan) on 2019-12-05 07:05:19 in reply to 22.1 [link] [source]

Deltification of forum post edits seemed a bit too easy, but also appears to work:

https://fossil-scm.org/fossil/timeline?r=forum-edit-deltify

What follows is console session output from testing this change:

-- Original post:
sqlite> select rid from blob where uuid='b1ad35942e3443d842a0eb8a4de04d5f524d38ca3f4a04d66773297f691d3892';
5852

-- Edited copy of that post:
sqlite> select rid from blob where uuid='c87823916cc8fe84aaef18ebeac5e2d0957bd5f99200c64fc1682466aa622d58';
6426

-- Was a delta created?
sqlite> .header on
sqlite> select * from delta where rid in (5852,6426);
'rid','srcid'
5852,6426

-- Interestingly, the addition of the G & P cards in the edit
-- make it larger than the original, even though the W-card
-- (post content) of the edit is 47 bytes smaller.
sqlite> select rid,size from blob where rid in (5852,6426);
'rid','size'
5852,1201
6426,1286

-- The above is the uncompressed/undeltified length. We're
-- expecting that its real blob.content strlen is smaller:
sqlite> select length(hex(content)) from blob  where rid=5852;
'length(hex(content))'
336
-- i.e. 168 bytes
-- Is there a non-C way to extract the human-readable string
-- (not hex-encoded blob) form of blob.content, i.e., the raw delta?

-- Uncompressed/undeltified content length matches expectations:
sqlite> select length(content(uuid)) from blob  where rid=5852;
'length(content(uuid))'
1201

# fossil artifact rid:5852 | wc -c
1201

It "seems to work", but another set of eyes on this change would be much appreciated.

(27) By Richard Hipp (drh) on 2019-12-05 12:21:58 in reply to 22.1 [link] [source]

Yes - it's that easy. Just identify the two artifacts that are candidates for delta compression, then invoke content_deltify(). The hardest part is usually figuring out which two artifacts are sufficiently alike that the delta is worthwhile.

Also, when doing deltas, it is best to try to figure out which artifact is likely to be accessed most frequently, and make that one the delta. So, for example, the most recent check-in is normally full-text and prior check-ins are deltas, under the theory that accessing the tip of the tree is more common.

Normally, in a tree with multiple branches, each branch is full-text. But when running "fossil rebuild --compress" or "... --compress-only" it does the extra step of trying to make the tip of each branch a delta off of the tip of trunk.

If you run a timeline with query parameters "?showid&v" it will show the RID values for the Manifest of every check-in and for every file. When the artifact is a delta, it shows the delta-source to the right of a left-hand arrow. So you can see how the compression goes.

There might be additional opportunities to do even more delta compression in the block chain. For example, there is no reason that two artifacts that are delta-compressed have to represent the same object (the same file, for example). If a single check-in has two files that are very similar to one another (for example, skins/blitz/css.txt and skins/blitz_no_logo/css.txt in the Fossil source tree) then it would be acceptable for one to be stored as a delta of the other. The tricky part is automatically finding such similar files. I've never tried to do that.

The content_delta() routine automatically takes steps to prevent delta-loops, which would be deadly to the repository - losing information. Also, the pre-commit checks that run on the repository prior to committing changes loop through and double-check that no delta-loops have been introduced. So there are safety mechanisms in place to prevent problems. Even so, you should be careful to avoid delta loops in the first place.

(28) By Stephan Beal (stephan) on 2019-12-05 13:01:20 in reply to 27 [link] [source]

Yes - it's that easy. Just identify the two artifacts that are candidates for delta compression, then invoke content_deltify(). The hardest part is usually figuring out which two artifacts are sufficiently alike that the delta is worthwhile.

In the case of editing a forum post, it's probably fair to assume that the previous version is the closest content match we're likely to find. Please see this trivial change, which implements that, at your convenience. It "works for me" :), but it seemed "a little too easy," and that's a bit disconcerting :/.

(29) By Warren Young (wyetr) on 2019-12-06 15:42:13 in reply to 28 [link] [source]

This forum post deltification feature could support the wished-for thread forking feature: you want the two versions of the post at the fork point to have different artifact IDs so that you can refer to both ends of the fork, so instead of having Fossil refer to the same artifact from both ends, forking a conversation could create a new copy of the old post's content and deltify it to get a very small on-disk artifact.

Doing it that way then opens the option for the two versions to drift: editing one wouldn't edit the other. I'm not sure if that's a good thing or not.

(33.1) By ss48 on 2019-12-08 13:38:10 edited from 33.0 in reply to 1 [link] [source]

It may also be useful to allow users with appropriate privileged to attach files or add pictures in individual posts.

Although subscribing to topics via email is possible, it would also be nice for user-specific inboxes in fossil (accessible to the left of the user name in the upper-right hand corner) that behaved similarly to an email inbox and let you manage unread check-ins, forum posts, and replies to issues from within fossil.

(35) By Stephan Beal (stephan) on 2019-12-08 08:10:38 in reply to 33.0 [link] [source]

Any per-user/per-thread state like that will likely blow up in our face at some point. Consider...

Flags such as "unread" and/or "read" would need to be set for every single post and every single user. The vast majority of forum users are idle, never logging in (this can be seen by their last login time in the admin interface), and the number of users steadily grows. At point we may well end up in a state where the forum db is often locked because it's busy updating the "unread" flag for for 5k users for each new post. For this particular forum the number of users will almost certainly always stay under 1000 (currently it's about 250), and its traffic relatively low, but the sqlite forum, once that project migrates from mailing lists, may easily be thousands of users generating far more posts than this forum does. 1000 users times (say) 20 posts per day means 20000 "unread" flags per day, the overwhelming majority of which which will never change because most people never read posts or do so via email. In only 100 days we'd hypothetically have two million such flags, each flag taking up a record in the database.

Recording a "is-read" flag instead of "is-unread" would be significantly cheaper because only a minority of subscribers are active, but still expensive and ever-growing in the long run.

For consistent forum behaviour, and backup purposes, those flags would need to sync to users who clone the repo, so anyone who clones it would suffer that bloat.

Since this forum has only a single sqlite3 db file behind it, data bloat is not a problem we can eventually simply throw more database servers at like we could in the a clusterable db environments which back full-fledged forum products. The long-term effects of any non-critical features which have the potential to cause "uncontrolled growth" need to be seriously considered before implementing them.

(37) By ss48 on 2019-12-08 13:37:22 in reply to 35 [link] [source]

I agree that syncing flags could be problematic. However, I think some alternate implementations might be possible and not as overwhelming to the fossil database.

Fossil already stores information about the user who posted to the forum. Fossil could provide users a view of the forum that just shows replies to the user's posts, or posts since the last post by the user. Those would not require any additional syncing as there are no flags to worry about.

There are instances where Fossil keeps data from going to the main repository. For example, https://fossil-scm.org/fossil/doc/trunk/www/private.wiki, fossil does allow creation of private branches only available in the repository cloned by the user. Similarly, if a user cloned a repository, the above information could be stored on the local clone of the database, and only accessible on that device. I think this approach (that takes advantage of the distributed nature of fossil) could work very well for storing user-specific settings, and can be a good solution for "uncontrolled growth".

Also, fossil is not just a sqlite database, but also an executable. As part of an executable, a separate database could be created with a separate web interface for subscribing and monitoring discussions for multiple fossil projects.

(36) By anonymous on 2019-12-08 12:44:48 in reply to 1 [link] [source]

It might be useful to support attachments for the bazaar model.

https://board.asm32.info/bazaar-model-with-fossil-scm.269

(38) By ss48 on 2019-12-08 13:44:31 in reply to 1 [link] [source]

One other feature. It would be useful to be able to allow moderators of the forum to sticky a forum posts, so that discussions related to a new releases or guidelines for posts could be kept at the top for forums that have a high volume of posts.

(49.1) By blissend on 2020-01-10 17:36:37 edited from 49.0 in reply to 1 [link] [source]

Collapsible Hierarchical View

Love the hierarchical view in general and to expand upon it, it would helpful having another way to filter through comments more by collapsing the parent. As indicated in https://en.wikipedia.org/wiki/Conversation_threading the potential advantage is even more elimination of time constraints.

Eliminates turn-taking & time constraints - Threaded discussions allow readers to quickly grasp the overall structure of a conversation, isolate specific points of conversations nested within the threads, and as a result, post new messages to extend discussions in any existing thread or sub-thread without time constraints. With linear threads on the other hand, once the topic shifts to a new point of discussion, users are: 1) less inclined to make posts to revisit and expand on earlier points of discussion in order to avoid fragmenting the linear conversation similar to what occurs with turn-taking in face-to-face conversations; and/or 2) obligated to make a motion to stay on topic or move to change the topic of discussion. Given this advantage, threaded discussions is most useful for facilitating extended conversations or debates [3] involving complex multi-step tasks (e.g., identify major premises → challenge veracity → share evidence → question accuracy, validity, or relevance of presented evidence) – as often found in newsgroups and complicated email chains – as opposed to simple single-step tasks (e.g., posting or share answers to a simple question).

Codeblock accept first line

When you write a code block with newlines the first line if next to the opening tag is omitted from preview. Example would be ```asdf and more new lines below it. The first line won't show asdf in preview.

However the last line followed by closing markdown tag qwer``` would be seen in preview. This is nitpicking by me and may be more trouble than it's worth.

Topic labeling

Seeing others not a fan of categories for the forum topics (which I understand) I thought an alternative would be labels on topics. It could also help with searching by allowing results to be filtered.

(50) By Stephan Beal (stephan) on 2020-01-10 17:46:31 in reply to 49.1 [link] [source]

Example would be ```asdf and more new lines below it.

The text immediately following the backticks is used only to tell markdown what programming language that block is for, and markdown applies gat as a CSS class called language-asdf. If we change that behavior, it's not only a notable difference from other markdown implementations, but it alao makes it impossible to implement proper syntax highlighting in those blocks ("proper" meaning being able to explicitly specify which language to use, rather than allowing/forcing the highlighter to guess via trial and error (which is, especially on short snippets, often wrong)).

(51) By blissend on 2020-01-10 18:13:36 in reply to 50 [link] [source]

Oh cool! I'm not as familiar with others but thought I read the codeblock section in https://fossil-scm.org/forum/md_rules well enough. If not there already might be worth mentioning 8)

(52) By Warren Young (wyoung) on 2020-01-11 01:06:19 in reply to 51 [link] [source]

I read the codeblock section in https://fossil-scm.org/forum/md_rules...

Because of its intentional brevity, that document has holes in its description of the Markdown syntax that Fossil understands. Contrast its size with that of the CommonMark spec, and you can see why.

Fossil Markdown doesn't implement 100% of CommonMark, but, well, let's just say that Fossil aspires to CommonMark compliance, so that if there's a discrepancy between the two specs, it's a good bet that Fossil actually does what CommonMark says.

I've enhanced md_rules to explain this better; it will appear online the next time drh builds and installs Fossil. You can build a local Fossil and check md_rules there in the meantime. Contrast CommonMark's treatment of the same topic.