Fossil Forum

Seeking feedback on difficulty / suitability of my Fossil wishlist
Login

Seeking feedback on difficulty / suitability of my Fossil wishlist

Seeking feedback on difficulty / suitability of my Fossil wishlist

(1) By patmaddox on 2023-04-16 20:40:10 [link] [source]

Greetings! I am a new, enthusiastic Fossil user - with a few itches :)

I have assembled my Fossil wishlist and am asking for guidance from experienced Fossil devs on how to tackle it.

Specific questions I have for the items I've marked with my interest and perceived difficulty:

  1. Can Fossil already do the thing I'm asking for, and I just don't know how?
  2. How difficult do you think it would be for a first-time contributor to pull off?
  3. How do you see it contributing to or detracting from Fossil's goals / purpose?
  4. Can you think of any pitfalls I should be aware of?
  5. What would you suggest as the immediate first action (e.g. look at some specific source file)?

Any information you can provide will help me prioritize them and get to work. Thank you in advance!

(2) By Stephan Beal (stephan) on 2023-04-16 22:44:31 in reply to 1 [source]

I have assembled my Fossil wishlist and am asking for guidance from experienced Fossil devs on how to tackle it.

Markdown checkboxes: because such state cannot be modified dynamically, i'd recommend against using form elements and instead use the modern Unicode emoji which were created for that, which obviates the need for any new parser-side functionality:

  • ✅☑✔

i think it's fair to say that we general don't want to extend markdown beyond what's absolutely needed for fossil-specific functionality, e.g. linking to artifacts by their hash. Doing so may well bite us in 5 or 10 years when markdown evolves to make use of syntax for which we've applied different semantics.

Auto-redirect docs on 404 ... Fossil will have a record of any moves that happen, so requesting a doc should 301 to the current location.

That would probably be straightforward in an SCM which did not have branching, but a redirect would need to know which branch to redirect to. If we visit a renamed doc 3 years after it's been moved, a lot of history may have changed, the branch(es) involved may have been renamed, etc., so it would not be straightforward to formulate a new /doc/XYZ part of the link. i won't claim it's impossible, because i normally get proven wrong, but my instinct is that it would be difficult to implement properly. For one, such a redirect could well leave the user browsing a different branch than the one they were initially on, potentially leading to much confusion.

Rather than "silently failing" by redirecting to what might be the proper version of the doc, reporting a 404 sounds like the right thing to do to me. User friendly? No. Correct? Arguably so.

Note that moving/renaming will always have corner cases revolving around recycled names. e.g. if file A is renamed to B and file C is renamed to A, chaos ensues.

Doc links to dir / files using $CURRENT ... Potential snag: files on disk may not have been added to the repo. How will the file explorer handle this?

The special doc branch name "ckout" is used for browsing docs from the current local checkout with the "ui" command. That name evaluates to the same as "tip" in all other contexts. "ckout" can be used to read files which have not yet been checked in, and that is often used when creating/editing new embedded docs.

When you use relative paths in doc links, the current branch is assumed, so $CURRENT is only rarely useful. Specifically, it's needed when using absolute links from embedded docs. In practice, however, absolute links to other docs are rarely needed.

(Edit: you've apparently discovered ckout already, as noted below.)

Annexed files

No idea what that means. You lost me at "git" ;).

One big question: how many files can Fossil handle? Git is slow when you get to ~200k files or more.

There is no absolute value on that, but certain aspects of fossil perform poorly with large numbers of files. Fossil is built for small-/medium-sized projects and its data model does not scale well to files with hundreds of thousands of files. Possibly the largest current active fossil repo is TCL core:

$ f dbstat -R tcl.fossil
project-name:      Tcl Source Code
repository-size:   392,642,560 bytes
artifact-count:    211,542 (stored as 66,985 full text and 144,557 deltas)
artifact-sizes:    53,055 average, 3,583,312 max, 11,223,454,970 total
compression-ratio: 28:1
check-ins:         34,158
files:             3,753 across all branches
wiki-pages:        59 (425 changes)
tickets:           8,240 (50,397 changes)
events:            1
tag-changes:       2,513
latest-change:     2023-04-16 18:36:18 - about 0 days ago
project-age:       9,153 days or approximately 25.06 years.
project-id:        1ec9da4c469c29f4717e2a967fe6b916d9c8c06e
schema-version:    2015-01-24
fossil-version:    2023-03-02 17:13:07 [27c3423e89] [2.22] (clang-14.0.0 )
sqlite-version:    2023-04-10 18:44:00 [4c5a3c5fb3] (3.42.0)
database-stats:    47,930 pages, 8192 bytes/pg, 1 free pages, UTF-8, delete mode

Additionally, fossil's internal structures have a hard-coded limit of just shy of 2GB for individual blobs, so it cannot handle huge media files and such. Changing that, such that all relevant code behaves properly with larger blobs, would require a "significant effort."

At some point someone had the BSD "pkgsrc" repository in fossil, but that's a pathological case for fossil and it cannot be recommended for such massive repositories.

(Maybe) Show / link to history in docs

Certainly feasible but has so far not proven necessary.

(Maybe) Render dir markdown as docs ... Or more likely support the ?ci=ckout special checkin.

Feasible, certainly, but care must be taken to be 100% certain that such a feature could not access anything outside of a check-out tree (e.g. it must not follow symlinks). With very few exceptions, the web interface does not access files directly, instead being sandboxed to the repository and checkout databases.

I want to link to the dir page so I can see the file structure.

Your local OS's file manager already does that far better than fossil ever could, so i'd argue that adding ckout capability to /dir would be superfluous.

Any information you can provide will help me prioritize them and get to work.

Be aware that fossil does not accept non-trivial patches without a copyright waiver on file with our project lead. See src:/doc/trunk/www/contribute.wiki for complete details.

(3.1) By Warren Young (wyoung) on 2023-04-16 23:31:47 edited from 3.0 in reply to 2 [link] [source]

Markdown checkboxes…use the modern Unicode emoji

Agreed, unless someone can point to a widespread standard.1 Checkbox markup isn't in CommonMark yet, the single best place if you want to bring pressure that $FEATURE should also be in Fossil; second-best is for it to appear in multiple other MD interpreters. With the exception of the Fossil-specific things you bring up, I agree, we shouldn't be blazing new MD syntax.

it would not be straightforward to formulate a new /doc/XYZ part of the link

While it is indeed not straightforward, drh did the work already; he needed to extend SQLite's capabilities in order to manage it! Seek out the "WITH RECURSIVE clade" code in src/finfo.c.

My own wishlist item is for that to be ported to the "fossil finfo" CLI interface; currently it applies to /finfo alone.

I do believe following a file's clade will yield @patmaddox's desired effect.

Possibly the largest current active fossil repo is TCL core…At some point someone had the BSD "pkgsrc" repository in fossil…

It's still there, and by the timestamps, appears to be an active mirror. They've also got the history of the NetBSD OS proper in Fossil. These appear to be pure mirrors, not active development tools, if I can judge by the lack of a home page, wiki articles, skin customizations, etc., but I'm an outsider to the project, so maybe they've quietly shifted over and not told anyone here.


  1. ^ GitHub counts as a "standard" in about the same way Windows did in decades past: a single idiosyncratic implementation that merely happens to have captured a large market share.

(6) By patmaddox on 2023-04-17 01:51:50 in reply to 3.1 [link] [source]

Checkbox markup isn't in CommonMark yet, the single best place if you want to bring pressure that $FEATURE should also be in Fossil; second-best is for it to appear in multiple other MD interpreters.

I am backing off tasks lists slightly, because from a practical standpoint I can use unicode, TODO/DONE, etc. So in the short term, it's not going to make a huge workflow difference.

However - we use git at $JOB and that's not going to be changing any time soon. I use these checklists all the time, and it would be nice to have them in Fossil as well.

MANY services support GitHub-flavored Markdown these days. As far as multiple other MD interpreters goes - this list indicates that the syntax I shared is supported by GFM, ExtraMark, Showdown, Gitlab, Haroopad, and iA Writer. That's from a couple years ago, too. Presumably all using GFM, as I think it's the only major language variant that defines it. Now, Pandoc supports it.

So it's not in the standard, but it's hardly niche. It's an elegant syntax, I'm hard-pressed to think that anything else will top it.


A more pertinent question: why not make the markdown parser configurable? It's already configurable between wiki and markdown. GitHub releases cmark-gfm and the license shouldn't pose any issues. There are 129 million README.md on GitHub... when people import their repos to Fossil, it would be nice if the README looked right.

(7) By patmaddox on 2023-04-17 02:17:54 in reply to 2 [link] [source]

That would probably be straightforward in an SCM which did not have branching, but a redirect would need to know which branch to redirect to. If we visit a renamed doc 3 years after it's been moved, a lot of history may have changed, the branch(es) involved may have been renamed, etc., so it would not be straightforward to formulate a new /doc/XYZ part of the link. i won't claim it's impossible, because i normally get proven wrong, but my instinct is that it would be difficult to implement properly. For one, such a redirect could well leave the user browsing a different branch than the one they were initially on, potentially leading to much confusion.

I'm admittedly not sure what it means to move a file to a different branch. But, I'm proposing that Fossil looks for any move history in the same branch as what was requested.

I guess one thing I'm not familiar with is the notion of "tip" - which I think is always the most recent check-in for a path, regardless of which branch it's on?

In any case, I am talking about the case where someone visits /doc/trunk/www/foo.mdand it has since been renamed to bar.md.

fossil mv records a name changed event. It's easy to spot with fossil timeline -p foo.md which will show the commit that renamed it.

Rather than "silently failing" by redirecting to what might be the proper version of the doc, reporting a 404 sounds like the right thing to do to me. User friendly? No. Correct? Arguably so.

I argue that that is exactly incorrect. HTTP 301 exists for this precise reason. "That thing you requested at /foo can now be found at /bar." It may have even later been moved to /baz, in which case there would be another 301 indicating that.

Note that moving/renaming will always have corner cases revolving around recycled names. e.g. if file A is renamed to B and file C is renamed to A, chaos ensues.

Eh... I'm not sure about that. In that case, A exists at the path requested - so Fossil should serve it up without a second thought.

(10) By Stephan Beal (stephan) on 2023-04-17 03:27:45 in reply to 7 [link] [source]

I'm admittedly not sure what it means to move a file to a different branch.

You can't do that but you can (effectively) rename ("move") branches. Any given version of any given file can exist in any number of branches at once.

To throw a wrinkle in it, fossil permits any number of branches (related or not) to share the exact same name. When resolving such a name, it will always select the one with the most recent checkin.

I guess one thing I'm not familiar with is the notion of "tip" - which I think is always the most recent check-in for a path, regardless of which branch it's on?

Correct.

... chaos ensues.

Eh... I'm not sure about that. In that case, A exists at the path requested - so Fossil should serve it up without a second thought.

Okay, "chaos" may well be overstated, but cases such as that one muddle the SCM history and introduce uncomfortable questions about how humans should interpret the history.

(8) By patmaddox on 2023-04-17 02:33:16 in reply to 2 [link] [source]

The special doc branch name "ckout" is used for browsing docs from the current local checkout with the "ui" command. That name evaluates to the same as "tip" in all other contexts. "ckout" can be used to read files which have not yet been checked in, and that is often used when creating/editing new embedded docs.

/doc does, /dir and /file do not. So, if make a doc link to /file?name=src/foo.c&ci=$CURRENT, then it won't work because 1) $CURRENT is apparently only supported in /doc/ links (2.2) and 2) /file and /dir don't support ckout.

Your local OS's file manager already does that far better than fossil ever could, so i'd argue that adding ckout capability to /dir would be superfluous.

It's not about using Fossil as a file browser while I'm developing, but about previewing what's going to be appear when I commit. I'm working on docs. I want to link to a file in the code base. I can preview the docs, but I can't preview what happens when I click the link. "It shows the code" - which is correct, but sometimes that code is Markdown, and I want to preview that as well.

Fossil already recognizes (wonderfully, I might add) that previewing the checkout content is useful to make progress. I don't see any reason why doc should be special in that regard over /dir and /file.

I can apply your same argument to docs: "any decent Markdown renderer already does that far better than Fossil ever could, so I'd argue that previewing /doc would be superfluous."

(9) By Stephan Beal (stephan) on 2023-04-17 02:56:33 in reply to 8 [link] [source]

/doc does, /dir and /file do not.

Those are 3 different things. /doc is specifically for rendering so-called "embedded doc" files, /file is for browsing the history and content of arbitrary SCM'd files, and /dir is (for the most part) a front-end for accessing /file links.

I can apply your same argument to docs: "any decent Markdown renderer already does that far better than Fossil ever could, so I'd argue that previewing /doc would be superfluous."

Not at all: fossil is the only such renderer which can render fossil's flavor of markdown and its own wiki format, e.g. its handling of links to hashes and wiki pages. There is no external tool which can do that for you, while there are hundreds of full-featured generic file managers.

Fossil already recognizes (wonderfully, I might add) that previewing the checkout content is useful to make progress. I don't see any reason why doc should be special in that regard over /dir and /file.

i'm apparently missing what you want to see there. Are you suggesting that /dir?name=ckout should list the contents of the checkout dir? If so, i stick by my suggestion that your file manager of choice is the far superior tool. If that's not what you're looking for them i'm apparently misunderstanding. It's common to have files in a checkout which are not part of the repository, and showing those in /dir would likely just cause confusion by giving the impression that they are in the repository.

(4.1) By Warren Young (wyoung) on 2023-04-17 01:37:35 edited from 4.0 in reply to 1 [link] [source]

Tag a range of commits so they appear as a single commit on the timeline

I believe you can get this in a Fossil way by first taking the advice you got in the other thread: use branches.

Given a series of incremental commits toward some broader goal:

  fossil ci -m 'first flailing hack at the feature'
  $EDITOR src/foo.c
  fossil ci -m 'less broken improvement on the feature'
  $EDITOR src/bar.c
  fossil ci -m 'approved version of feature'

…you can take all three commits and approximate a "git rebase" squash operation like so:

  fossil amend --branch my-new-feature ABCD1234
  fossil up trunk
  fossil merge --integrate my-new-feature
  fossil ci -m 'approved, squashed-flat version of feature'

The main tricky bit is finding that magic ABCD1234 hash, but a "fossil timeline" command will pull that up for you.1

This doesn't work so well if you have to tease apart multiple commits back in history, rather than be able to shunt the entire tip of a development branch off as shown above. You can manage it with a lot of "fossil merge --cherrypick" commands, however, which approximate Git rebase. You'd create a new branch, cherrypick the individual commits to it, then merge that down with --integrate instead.

All that's wanted now is one last feature: the ability to automatically hide the source branch. I believe the most Fossil way to get that is to add a "fossil merge --hide" option, which simply adds a hidden tag (same as "fossil amend --hide") to the root of the merged-in branch.


  1. ^ Alternately, open your graphical /timeline, find the commit, click on it, then click "edit" and put my-new-feature into the box marked "Make this check-in the start of a new branch named:" and hit Submit.

(5) By Stephan Beal (stephan) on 2023-04-17 01:42:06 in reply to 4.0 [link] [source]

You'd create a new branch, cherrypick the individual commits to it, then merge that down with --integrate instead.

To add a small bit to that: note that --integrate is only a shorthand form of merging and then "closing" the merged-in branch. If you fail to add the --integrate flag then the merged-in branch will remain "open", but can then be easily closed via the timeline (click on that checkin, then there's a checkbox to "close" it somewhere in there) or via the fossil branch close BRANCH-NAME command.

All that's wanted now is one last feature: the ability to automatically hide the source branch. I believe the most Fossil way to get that is to add a "fossil merge --hide" option...

The "branch" command also has hide/unhide subcommands. Adding such a flag to the merge command would be a bit more involved because it would have to queue up that flag for the pending commit. "integrate" is queued up by changing the vfile.chnged field, and perhaps "hide" could be done the same way but that would have to be carefully tested/audited for undesired side effects.