Prior discussion of PIC markup in Fossil documentation was at post db217184de. The EQN markup would work much the same way.

Since posting that a couple of months ago, I did actually attempt to write a PIC language interpreter in Javascript. But I discovered that Javascript is not a particularly good language with which to write a parser/compiler - or at least my limited knowledge of Javascript was insufficient for the task. Input from readers with better Javascript skills would be appreciated here.

I have heard that it is now possible to compile C code into WASM. In particular, I hear that people are compiling "sqlite3.c" into WASM and running it in browsers. Perhaps an alternative approach would be to write a compact PIC translator in C (using Lemon to implement the parser) then compile the result into WASM for download to the browser. But I don't understand how all of that works. If anybody has any insights and can help guide me through the process, that would be appreciated.

A question about lemon. IIRC it is a LALR parser generator similar to Bison.

Is the generated parser able to execute semantic actions as the AST is built ? This and the ability to reach back to the lexer to control its execution will be necessary for PIC.

The reason for that is PIC's macro system, which it has in lieu of proper procedures/functions. Macros can recursively call each other, and to not have this end in tears of infinite recursion this expansion has to be/is curtailed in code not executed, i.e. the untaken branches of if/else conditionals.

For the implementation of the language this means that semantic actions have to be executed as the input is lexed and parsed, and then reach back to the lexer with the outcomes of conditions to selectively skip/suppress macro expansion.

See pages 6-8 of the dpic docs.

With that in mind I wonder if it might be suitable to reuse an existing parser like dpic, instead of writing a new PIC parser, for compilation to WASM.

If you can compile dpic down into a dpic.js file that we can source and use to render PIC on the client-side, that would be great!

it is now possible to compile C code into WASM

Yes, and more to the point, WebAssembly support is approaching universal adoption. It won't run in ELinks, but this is a non-core feature that those with such niche needs can ignore.

I do wonder if server-side rendering to a virtual SVG would be a better plan, though.

Is there a place (someone's repo on ChiselApp?) where we could collaborate on the PIC renderer?

I seem to have created a Lemon grammar for PIC, but I cannot test it on real inputs yet, because first I'll have to hack the lexer to understand defines and substitute macros. (According to Kernighan and Raymond, they have to be done outside the parser, by pure sting substitution.)

Also, I haven't started the work on the "virtual machine" to interpret the AST and output SVG.

If you can wait about 12 hours i can set up a repo for you on fossil.wanderinghorse.net and make you the admin so you can manage collaborators. i have no idea whether the backoffice (forum) parts will work on that site, though, so no guarantees on that. If you want to go ahead and set up a repo, set up your user(s), and email me a link (see https://wanderinghorse.net/home/stephan) where i can download it, i can have it installed in no time tomorrow. My only requirement is that i be allowed to set myself up as an admin/superuser solely for maintenance purposes.

Alternately, i can set you up an empty repo. Just email me (see previous post) with your preference.

The only skeptic in the room

But you are not alone!

You will notice that version control is not on either of those lists.

That's... kind of mind-blowing. Perhaps they consider it an implicit entry in the list, since it's a level of infrastructure which underpins every aspect of what they do. A "can't see the forest for the trees" kind of thing.

What limitations or restrictions in Fossil are limiting productivity?

My own personal entry in that list is not new: the lack of a library interface. Obviously (for those who don't know this tidbit of history) i intended to tackle that with libfossil, and was well on the way to feature-parity with the fossil binary when chronic RSI (specifically, C8 nerve inflammation/damage in both arms) effectively kicked me out of the game in late 2014 (even now, my typing capacity is still reduced to about 10-15% of what it previously was, and i am still in forced early retirement due to the years-long duration of my medical leave).

Coincidentally enough, since yesterday i've been milling over a reboot of that work, focusing initially on features which would make it easier/possible to create custom read-only clients, ignoring the ability to update/merge/etc. for the time being, simply to try to keep the scope in check. The motivation for that came from user elechak, who posted a query based on the mlink table. Being able to plug that type of feature one's own app/page/whatever would empower all sorts of apps we don't/can't currently have. Obviously once we have a library, the gateway is opened for any number of clients, e.g. tooling for reports, CI, and IDEs.

That said... my hands cannot reasonably commit (as it were) to that scope of project for the foreseeable future :(. Getting the library up and running again, now that fossil supports multiple hash types (libfossil predates those changes), would require an as-yet-unknown amount of effort.

That said, i don't think it would take too awful much to get the read-only functionality working again. So... i'll take this opportunity to bang the drum again and call for "seriously interested" coders to get in touch and help make that happen. (Forewarning: it wouldn't be a small amount of code.) While i cannot personally commit to more than "casual" amounts of coding, my capacity for technical support is far less bounded and my capacity for moral support is practically limitless ;).

One GitHub-er asked me: "What would it take to get SQLite to move off of Fossil and on to Git."

LOL! i haven't laughed this hard since you explained the logic behind you getting taller with age!

i intended to tackle that with libfossil

Let's not make the same mistake twice: libfossil fell into disrepair because it's a second implementation of the formats and protocols within an ecosystem that's not able to progress as fast as it wants to on a single implementation. (Thus this thread.)

Rather than resurrect libfossil, I'd rather see that work went into refactoring the internals of Fossil to extract a library, divorcing the CGI and CLI interfaces from a core that only deals with the repo formats.

Some of this is already done well enough that all that's needed is to draw the lines more explicitly, but I want to get to a world where the build process results in a static libfossil2.a that links to the Fossil binary, which can optionally be built as a shared object and linked into an app without linking to libnss, libssl, etc. because the UI is a native GUI tool that only works on local repos and thus doesn't need network clone and sync support.

The resulting library should be modular, so that if the third-party app also wants clone and sync support, it can include that as well.

Or, Plan C: expand the current web APIs to the point that one could write a full-function Fossil library that did nothing other than make HTTP API calls to a running Fossil instance.

Proposed Forum Enhancement

I would like to start a new Forum thread named something like "Libfossil Reboot" based on the previous comment. The new "Libfossil Reboot" thread would link back to Warren's post, and Warren's post should link forward to the new thread. But it would be a completely new thread, in as much as it is a tangent off of the original.

In other words, I'd like to create a simple mechanism to prevent thread hijacking.

Somebody with good UX sense, please suggest a suitable interface for this new feature, or if the new idea makes no sense from a UX standpoint, please talk me out of it.

At risk of creating a hijack of a hijack (🤷‍♂️) what if a Moderator or higher level user could mark a post as being a topic fork, which then prompts the user for a new thread title. The existing post becomes the starting post of the new thread under the new title, and unlike current first-post-in-thread, it has an "in reply to" in its header pointing back to the other thread.

For normal users, all they see is that the Reply button changes to Moved or similar. Clicking it takes them to the start of the other thread, where they can now click Reply.

Don't send the user straight to a reply: they need to see the new context first. They might then choose to reply further down-thread in the child topic.

(Edit: sorry, Richard - i didn't see your note about the threadjacking until posting this :/. Further discussion will take place in the new thread if/when it starts.)

Fair enough.

Rather than resurrect libfossil, I'd rather see that work went into refactoring the internals of Fossil to extract a library, divorcing the CGI and CLI interfaces from a core that only deals with the repo formats.

That's more or less originally what the goal was. There are, however, fundamental design decisions in fossil which make "bending it" into a library unfeasible. First and foremost is its allocation failure policy: simply die. That drastically reduces the amount of code needed (as can be seen when comparing the same algos in fossil core to libfossil - a huge number of lines of code in libfossil are checking/handling result codes and allocation results, where fossil would simply die in most such cases). A library cannot feasibly have a die-on-OOM policy. (That is to say: i wouldn't personally use a library which i knew would outright kill my app if the library encountered an error code or couldn't allocate memory.)

That said: i would be completely on-board with such an effort. As Richard put it when it was proposed it in 2011, though, it would require "a Herculean effort." (Though dictionaries/style guides differ on whether Herculean is capitalized or not when used that way, it would certainly have a massive capital H in our context!)

Or, Plan C: expand the current web APIs to the point that one could write a full-function Fossil library that did nothing other than make HTTP API calls to a running Fossil instance.

That's kinda/sorta what the JSON bits do, and an HTTP-based API is essentially a link-at-call-time DLL. However, a JSON-based API would be a particularly poor choice for certain functionality, most notably transferring binary files of arbitrary sizes, as doing so requires encoding into a form JSON can use (huge strings or massive arrays of integers (byte values), both of which are larger than the binaries themselves).

Understand, I don't mean that your intent in creating the library was a mistake, only that we now know that it clearly didn't attract a big enough user community that you had to fend off the tendency to fork the libary when you had to give up on it, because you had multiple people all wanting to continue it. Knowing this, we shouldn't repeat the decision.

Contrast Fossil: I can see this project forking if something drastic happened to drh or to the project direction. We've got enough people actively interested in working on it that we would have to fight off a tendency to create multiple forks and not just one new successor project. I point that out not to show a danger to Fossil, but to show that it has a bus factor above 1.

Years now after libfossil stopped moving, I think we can say that it had a bus factor of exactly 1. :(

Therefore, efforts going forward should make Fossil itself more powerful, not divide precious resources.

As to the matter of difficulty, that's why I proposed my previously-unnamed Plan B as a refactoring effort, rather than reimplementation. It's something we can simply choose to strive toward, one bite at a time.

On the efficiency of a JSON API, many HTTP APIs switch content types to match the needs of the call. JSON returns may be the default, but binary and HTML returns are also legal based on context. Some calls may even have multiple forms. In that sense, Fossil UI is the HTML API to this library already, and /raw is the binary interface. All I'm proposing is wrapping this in a way that you can use inside, say, a Fossil plugin for a programmer's text editor.

Git is used by nearly everyone and is the defacto standard, and so that is what they use.

That's pretty much the "Windows" argument. Windows hasn't gone away in the face of smartphones, Chromebooks, and the cloud, but it has been pushed to the margins wherever practical. It's probably a large part of the reason that the Windows phone and tablet efforts didn't go anywhere.

It's the point of view of an entrenched victor who becomes blind to the fact that they are now entrenched, and consequently can no longer move.

I'm not even sure this is a conscious choice. Perhaps a better metaphor is Gulliver strapped down by thousands of Lilliputians using ropes woven from process, standardization, and interdependency. Gulliver could choose to rise, but he'd destroy all the ropes in doing so.

1. Poor tracking of files across renames

That's not the number one issue, just the most recent request. :)

Fixing this would be a nice quality of life improvement for those of us already using Fossil, but I doubt it factors very high in anyone's decision about whether to use Fossil, or whether to leave Fossil for something else.

2. Inability to do shallow clones

That goes to scalability, and it probably does have a lot to do with Fossil's adoption curve.

It is true that most software projects aren't Linux-scale in terms of SLOC and complexity of development and thus don't benefit from elements of Git's design that serve the rare characteristics of the Linux kernel development project.

Nevertheless, it is also true that there are a lot of software projects bigger and more complex than SQLite yet still much smaller than Linux and thus could benefit if Fossil supported large-scale projects better.

3. Poor ticket UX

Now that we have Markdown in tickets (!!) my only serious complaint about it is the two-step filing process.

I see the value of the simplified filing interface for normal users, but when I'm logged in as a Setup user, it's annoying to not have all of the ticket fields available to me to fill out at the same time as the initial filing.

This could be controlled by a user cap saying whether that user or category of users sees the full ticket UI on filing.

Speaking of user caps, another thing I anticipate is support for various SSO technologies: LDAP, OpenID, Oauth, SAML, etc. That in turn would require some way to map groups in those systems to user categories in Fossil, which would probably have to expand beyond the current hard-coded set of four.

4. Inability to subscribe to notifications for changes to specific tickets or files.

That sounds like it'd be pretty easy to implement. All it wants is someone to scratch the itch.

Not me; don't care. :)

5. Difficulty implementing actions that are triggered by new check-ins and/or new tickets.

Server-side Tcl hooks?

I think the hardest part is making such hooks enforcing in the face of distributed clones. If I check something into my local repo but a push to the parent repo fails because of a Tcl hook refusing the new material, what then?

LLVM...downstream tools...stuck with having to rebase everything.

Wouldn't a release branch (as opposed to release and version tags) solve this? Downstream projects could point at the branch they care about, which would get only a linear sequence of changes, pushed there by a release manager.

You can implement development, feature freeze, slush, and code freeze branches in this way within Fossil today, progressively restricting changes until a frozen blob of code lands on the release branch.

The only thing that might be nice in this regard is locking branches to particular users. Fossil's DVCS nature makes absolute locking impossible, but part of such organized thinking is an assumption that the users aren't actively malicious. If low-level Dev A checks something into the feature freeze branch directly, Fossil already has tools to fix that, and beyond that, it's a social/admin issue, not a tech issue.

GitHub is moving away from being a Git Repository hosting company

They're just moving up-stack, which is necessary to avoid becoming another commodity provider of Git service. If they'd continued to focus on Git, they'd be in direct competition with Digital Ocean and /usr/bin/git. They have to move up-stack to continue propping up those Silicon Valley salaries.

Fossil doesn't have the same need. It merely needs to remain useful to its users. I think this discussion should remain focused on that and possibly on attracting new users. But Fossil doesn't have the same business needs as GitHub, Inc., a Subsidiary of Microsoft Corporation.

If I check something into my local repo but a push to the parent repo fails because of a Tcl hook refusing the new material, what then?

What happens when you push to a parent repo where you don't have commit privilege?

I see where you're trying to go with that point, but the difference is that when you try to push without check-in capability it's expected to be permanent until you get an account with that capability, then permanently allowed as long as your account remains active. To compare that with the denial from hooks, you're saying that a remote piece of software can effectively activate or deactivate my check-in capability on each artifact pushed.

Keep in mind that Fossil writes its check-ins durably first to the local blockchain, then syncs the result to the remote system. This means that once local artifacts are denied by the remote, they'll be denied until the remote changes its rules.

If we assume that the rules implemented by the Tcl commit hook are good and wisely crafted, then this means the local repo will forever have durable artifacts that can never be pushed. The local repo will keep pushing them, and the remote will keep running the commit hook logic over them, which will keep rejecting them.

All of this is fairly anti-Fossil, which tries very hard to make all clones identical.

If Fossil gets anything like this feature, I think it must be done by the hook scripts being somehow sent down with the clone and kept up to date there, so that the hook can run locally before the local blockchain gets modified.

I think it must be done by the hook scripts being somehow sent down with the clone

...which of course introduces portability problems: you have to ensure that the hook scripts will in fact run correctly on all of the clones, else you break those clones. "But it worked on my machine!" doesn't fly when your remote contributors stop being able to check work in just because they're on a different host OS, or are missing some third-party Tcl package, or whatever.

However, a night's sleep and a shower have provided an answer: instead of the hook script outright rejecting "bad" checkins, what if Fossil treats commit hook rejection as a signal to automatically create a branch for the new artifacts so that other work can continue on the original branch? Then the problem can be corrected in public, which is very much pro-Fossil.

We could obviously do this in a post-facto way, running such problem-detecting scripts on the server and having it automatically move artifacts, but making it part of the sync protocol with integrated hook scripts means the remote clone could be notified immediately that their check-in has been moved to a branch. The hook script could even return a detailed message explaining the problem, which could guide the remote user to a fix.

This new scheme means the content is never lost, it is committed locally and sync'd remotely, but it's set aside until the problem can be resolved.

what if Fossil treats commit hook rejection as a signal to automatically create a branch for the new artifacts so that other work can continue on the original branch?

I like this idea a lot.

but making it part of the sync protocol with integrated hook scripts means the remote clone could be notified immediately that their check-in has been moved to a branch.

I would hope that the hook script would be able to supply the new branch name, "calculated" by whatever rules the core project team deems appropriate.

Maybe the hook script could return a string with a branch name and a message separated by some reasonable delimiter.

the hook script would be able to supply the new branch name

Sure. It'd allow a form of namespacing, where each hook script gets its own branch naming prefix so you can tell which script shunted the commit aside. Examples:

1. code-style-abcd1234: code rejected by a site-configured automatic code style checker

2. ci-cd-abcd1234: this commit breaks the CI/CD build; implicitly an async hook

3. temp-code-abcd1234: this commit appears to contain some sort of temporary code: debug logging, FIXME comments, #if 0 blocks, etc.

Maybe the hook script could return a string with a branch name and a message separated by some reasonable delimiter.

Tcl lets you return a list of values; it isn't limited to returning a single scalar value:

    return { 
         401,
         "code style check failed",
         "code-style-$nonce",
         $output_from_style_checker
    }

That is, an error code, a brief message, a branch name to move the commit to if not null, and an optional long message that expands on the brief message.

I think "rebase" may be useful to entities who want to remove or hide errors or worse.

If you develop on a private branch, then merge the final commit to trunk then no one need ever see any mess there might be on the branch.

(Down side: Fossil doesn't track merges from a private branch, not even in the repo where the private branch exists.)

Thank you, I didn't know that.

Reminder to moderators:

We have had one spammer today who's tried to sneak in hyperlinks to a "bathroom mold removal" product as a sneaky link on a single period, making it effectively invisible when reading the markdown-processed post but clearly visible in the unparsed content. Please always check the unparsed content of posts for new and anonymous users before approving them. This particular piece of spam was first posted by a new account (which i deleted) and then again, later in the day, as an anonymous post (which snuck past moderation but i coincidentally recognized as identical to the previous post, so deleted it).

The devious part is that such posts appear to be topical, but they try to sneak in spam links which are easy for the eye to overlook.

This quote (from Tom Kremenek of the Swift project) is saying that your workflow will be defined and constrained by the tools you use.

Might you be talking about Ted, the Swift Project Lead?

Yes, it seems likely that I wrote his name down incorrectly in my notes. Sorry about that, Ted.

I'd like to copy this post for a personal blogpost (of course with reference) is such doing welcomed and permitted?

No objections from me.

One GitHub-er asked me: "What would it take to get SQLite to move off of Fossil and on to Git." Just to be clear to everyone reading this: That will never happen. Fossil was specifically designed to support SQLite development and does so remarkably well. Fossil fills a different niche than does Git/GitHub. Fossil will continue to be supported and enhanced by me (as well as others) well into the foreseeable future.

You could counter ask him/her: "What would it take to get full support to Fossil inside GitHub and make it a first class citizen there?" ;-)

Cheers,