libfossil Notable TODOs

This page gives a high-level overview of the notable TODOs, or perceived TODOs, as well as non-TODOs (topics/APIs which are either out of scope or are way, way down the line).

Core SCM (and closely adjacent) Features

In no particular order:

• vfile.mhash field: this was added sometime after the lib took a break and needs to be populated/handled by the lib code. (It seems to be only updated when merging?)

• files_of_checkin virtual table. This is needed for certain queries, e.g. fossil's /finfo page uses it.

• Port over checkout/repo fingerprint: this allows detection of when a checkout's repo has been replaced by one with different RIDs. See fossil's vfile.c:vfile_rid_renumbering_event(). Calculation and confirmation of the fingerprint was added on 2021-04-16 but still TODO is an attempt at automatic recovery.

• Rebuild: this feature exercises almost every major SCM feature of the framework except for merging and synchronization. Similarly, deconstruct/reconstruct might be useful for exercising the library. With the exception of missing ticket crosslinking, we have all of the pieces needed to implement this, and the f-parseparty test app does much of this work already.

• Cross-linking of tickets: the library can read and write them but they cannot yet be crosslinked. Currently, depending on a context-dependent flag, it may error out if it tries to crosslink one or it may silently skip over it.

• Double-check crosslinking of other types and make sure that we are not missing newly-added features.

• Stash support. First requires merge support. (Or does it just require update support? We have that part.)

• Unversioned files should be trivial to do.

• Maybe pending-moderation support (tickets, ticket comments, wiki edits), depending on how centrally-managed such data need to be (i.e. whether it can be delegated to an app layer).

Checkin

The SQL code related to checkins has diverged significantly from fossil since libf's checkin support was first implemented (the better part of a decade ago). Though libf's is known to work for the cases supported by its existing feature set, it "really couldn't hurt" to audit the checkin algorithm against fossil's. It's okay if they differ, so long as the results are the same. This will be especially important as merge support is added, since the libf checkin code has never dealt with that. What brings all this up is a corner-case discrepancy discovered in fossil's checkin support regarding merged-and-edited files:

https://fossil-scm.org/forum/forumpost/03f6b307f89c990b?t=h

for which Richard proposes a patch to fossil's checkin.c:

--- src/checkin.c
+++ src/checkin.c
@@ -2572,11 +2572,11 @@
   ** table. If there were arguments passed to this command, only
   ** the identified files are inserted (if they have been modified).
   */
   db_prepare(&q,
     "SELECT id, %Q || pathname, mrid, %s, %s, %s FROM vfile "
-    "WHERE chnged IN (1, 7, 9) AND NOT deleted AND is_selected(id)",
+    "WHERE chnged<>0 AND NOT deleted AND is_selected(id)",
     g.zLocalRoot,
     glob_expr("pathname", db_get("crlf-glob",db_get("crnl-glob",""))),
     glob_expr("pathname", db_get("binary-glob","")),
     glob_expr("pathname", db_get("encoding-glob",""))
   );
@@ -2610,16 +2610,18 @@
                    blob_str(&fname));
       blob_reset(&fname);
     }
     nrid = content_put(&content);
     blob_reset(&content);
-    if( rid>0 ){
-      content_deltify(rid, &nrid, 1, 0);
+    if( nrid!=rid ){
+      if( rid>0 ){
+        content_deltify(rid, &nrid, 1, 0);
+      }
+      db_multi_exec("UPDATE vfile SET mrid=%d, rid=%d, mhash=NULL WHERE id=%d",
+                    nrid,nrid,id);
+      db_multi_exec("INSERT OR IGNORE INTO unsent VALUES(%d)", nrid);
     }
-    db_multi_exec("UPDATE vfile SET mrid=%d, rid=%d, mhash=NULL WHERE id=%d",
-                  nrid,nrid,id);
-    db_multi_exec("INSERT OR IGNORE INTO unsent VALUES(%d)", nrid);
   }
   db_finalize(&q);
   if( nConflict && !allowConflict ){
     fossil_fatal("abort due to unresolved merge conflicts; "
                  "use --allow-conflict to override");

The corresponding code in libf looks much different than that now, and it's not currently (2021-09-17) clear whether or how that change would need to apply to libf's impl

Security-relevant

But not otherwise SCM-relevant...

• Port over db_unprotect() and db_protect_pop() APIs, which allow a db to effectively be made read-only except for limited windows where specific sections of it needs to be writable. Related: db.c:db_top_autorizer().

Non-SCM TODOs

In no particular order...

• Add SPDX-style license attribution to all source files. This is ongoing.

• Header file restructuring. The current separation of the APIs into many include/fossil-scm/*.h files is somewhat confusing. The initial intent was to keep my low-end development system of the time from choking on syntax highlighting on one large file, but those days are largely behind me. It may make sense to combine those into 1 public API file, 1 internal API file, and the auto-generated config file(s). (Even then, it's big enough to choke emacs' syntax highlighting on lower-end systems like Raspberry Pi SBCs.)

• Stop using char as booleans. This tree historically uses char type for booleans. Now that the tree is C99, we can switch to the bool type. This is ongoing.

• f-vdiff: port in fossil:3504672187af59f0 in order to be able to select the diff width based on the terminal size.

Maybe (and Maybe Not) TODO

• Undo support.

• Bring some of the configuration-related state up to date. The infrastructure is there but the exact list of fossil-supported properties is lagging behind by several years. On the other hand, the majority of such config state is really application-specific and probably has no business being handled by this library. Some settings, e.g. forbid-delta-manifests, we internally honor to avoid Grief in downstream repositories. Others, such as the various globs settings, have API-level support and can be honored by the appropriate library APIs via toggles (e.g. the file-add API optionally honors the ignore-glob setting).

• BOMs. Fossil's diff APIs internally convert their inputs to UTF8 and strip the BOM (if any) from them. libf does not do that. On the one hand i'm hesitant to do so because these blobs can be anything at all (not necessarily SCM controlled). On other other, for annotate's sake it might make sense to do so automatically because the user is passing in artifact IDs instead of file content. On the other other hand, the fossil routine for doing that (blob_to_utf8_no_bom()) is far, far more involved that simply stripping a BOM. i'm torn on whether that's the library's job or not, and really dislike having to either mutate the original inputs or reallocate them to make that conversion. OTOH, fossil does so.

• Symlinks. i have always strongly disagreed with the addition of symlink support into fossil: platform-specific constructs simply have no place in the core of any SCM (with the "effectively necessary," as well as unobtrusive, exception of the executable bit). For platforms which don't support symlinks, fossil stores/manages them as plain text files with a single line holding the name of the referenced file. This is very likely the only route the library will take to supporting symlinks, especially since the hassles symlink handling caused fossil in late 2020 (long story). Probably the only way the library will support proper symlinks is if someone who uses that feature adds and maintains it.

• Backlinks. Crosslinking "should" update the internal list of backlinks from certain text fields, but doing so requires parsing wiki/markdown-format text. See backlink.c in the fossil tree for the details. On the other hand, backlinks support only requires parsing wiki links, not the full grammar, so it might not be as painful as it initially sounds... though somewhat more for markdown, where we're required to do a multi-pass scan to handle its linking model. (We'd also need to handle verbatim blocks to avoid parsing links inside those blocks.)

• Ticket support. Ticket handling is surprisingly complicated, due largely to the customizability of the ticket database schema. If fossil-compatible ticket supports gets added to libfossil, it will very likely be because someone other than myself adds it! The core artifact data structure supports tickets, so the bits required for adding it are in place.

Optimizations

• Artifact parsing, in particular of checkins, is much slower in libfossil than fossil. Some of this is easily attributable to more abstraction layers, but certainly not all of it. Some optimization of crosslinking speed is certainly in order. As a point of comparison, try fossil rebuild vs f-parseparty --crosslink. libf parses non-checkin types "plenty fast," e.g. 1600-odd control artifacts in roughly 6ms on my main computer. Checkins, however: 6m20s for 15504 2m59s (debug) or 2m30s (non-debug) for 15504 checkins in the main fossil repo, as of this writing (just parsing, without crosslinking). On the sqlite3 repo it can only parse approx. 3000 checkins in 10 minutes, at which point that test got cancelled 24657 checkins in 10m9s (debug? Non-debug?). The reason for the serious speed degradation as the repo size increases is unclear.

  • This is at least partially (roughly 20-33%, based on basic tests) due to libfossil building in debug mode by default.
  • On 2021-03-24 the crosslinking was sped up by roughly 50% via the addition of a content cache identical to fossil's, but it still takes roughly 4m45s to parse and crosslink 16287 checkins in fossil's core repo with a debug libfossil build. 2m30s of that time is parsing.
  • 2021-10-03 update: a debug build of (f-parseparty -t c --quiet --crosslink) can load and re-crosslink the 2178 checkins in its own repo in about 4.5s. That same thing on fossil's 17099 checkins takes about 5m55s on the same machine.

• Buffer caching. The library internally has to use many temporary memory buffers. Some of those it reuses as much as it can (e.g. for filename normalization), but some operations (fsl_content_get() in particular) have to use several, potentially many, temporary buffers of arbitrary sizes, which can easily lead it to allocating hundreds of thousands of times for a total of 1GB+, in a single session even if it only allocates a max concurrent memory of less than 15MB. We can probably improve this situation by installing a buffer cache intended primarily for use with fsl_content_get(), in which we store some number of buffers totaling some certain max amount of concurrent memory. This could be achieved relatively inexpensively by either hard-coding a buffer array size (e.g. 10) or modifying fsl_buffer() to be a singly-linked list, the links being used solely for such a cache, and keeping the buffers in alloced-size order. The catch there, however, is that the decompression and de-deltification steps effectively makes reuse of such buffers next to impossible because we cannot easily and efficiently do those operations in-place in existing buffers.

Remote Synchronization

• This will(?) be implemented in terms of abstract streaming APIs, very possibly the ones the library already uses for the majority of its file I/O and abstracting output streaming (e.g. it uses an abstract output stream for diff output, rather than writing directly to a buffer).

• This will almost certainly be one of the last major features.

Wiki Parsing and Rendering

• This set of features hovers right on the edge of out-of-scope for the core libfossil. Rendering is necessarily output-format-specific and the library has no business defining such outputs. Also...

• The fossil implementations of these are not written with port-friendliness in mind, so a complete reimplementation would possibly be necessary.

• In order to support near-arbitrary applications the wiki parsers need to be implemented in such a way that clients can customize (via hooks/callbacks) how links/references are generated.

• The venerable Fossil Wiki format has the lowest priority. In practice markdown has easily taken the lead, and only old (pre-markdown) docs tend to be maintained in that format.

Non-TODOs

• Any and all UI-related elements, including HTML, CSS, and JavaScript. The library will enable such applications but will not provide, e.g., an HTML framework beyond perhaps (maybe) wiki rendering.

• Scripting of tickets. There are no current plans to 100% mimic fossil's th1 scripting of tickets. Fossil's use of TH1 was one of convenience, not long-term practicality. The library itself will have no "official" scripting language, but is designed specifically to make tying it to scripting engines relatively straightforward. We have a standalone copy of TH1 which "could" be integrated with little work, but doing so would not be an ideal path to go down.