Tip: seamlessly "merging" an independent documentation repo with its sibling source repo

(1) By Stephan Beal (stephan) on 2019-12-13 14:38:53 [source]

The Situation...

"i've got this repo" for which i'm experimenting with porting its 200 pages(!) of Google Docs-format documentation to markdown. Because this is an experiment, i don't really want to clutter up the main repo with tons of markdown files which i am just as likely to abandon/delete as i am to keep.

It turns out that the new "extensions" feature offers an elegant solution to this: in short, i can host the documentation in a completely standalone repository, but then embed its checked-out content under the main source repo using that feature's /ext path. This has a few implications:

Links in the embedded docs in the form [foo](/bar/baz) will refer to /bar/baz in the top-level repository. Likewise, links in the form [foo](./bar.md) will refer to /ext/bar.md in the top-level repository.
If i abandon this effort (porting 200 pages of accumulated docs is a bit daunting), no cruft is left in the main repo. Aside from avoiding "document corpses," many dozens, if not a couple hundred, commit entries might be avoided.
Profit! If i end up managing to port it, the docs can be checked into the main repo and all of the links still work because of point #1.

Mini-Howto

First, set up the "extension root" as explained in The Fine Manual. The extension feature automatically translates .md files found in the extension directory, effectively treating them like the embedded docs feature.

Secondly, place the in-development markdown files in that extension root directory, using subdirs for organization if desired.

Finally, browse to http[s]://your-repo.eu/fossil/ext/DOCUMENT.md.

That assumes that /fossil is your fossil server or CGI script (in my case, it's a CGI).

The DOCUMENT.md contains absolute-path links in the form shown above, they'll end up pointing to /fossil/ext/THAT/PATH, which is precisely what we'd like to have happen. Additionally, if links are in the form [label](./a-doc.md) then they will refer to /fossil/ext/a-doc.md, which is also exactly what we'd like to have happen.

The integration between the two disparate repositories is nearly seamless, in that links in the new documentation work in both the staging area (/ext) and the hypothetical final directory (/doc/foo in the main source repo). The one non-seamless part is that links from the source repo to the new docs must, of course, point to either /ext/... or /doc/..., as appropriate, but in my case the main repo will not have any links to the new docs until/unless they are migrated in to the main repo.

That's all there is to it.

(2) By Stephan Beal (stephan) on 2019-12-13 14:50:58 in reply to 1 [link] [source]

Follow-up: there's one embedded docs feature which doesn't apply under /ext which "would be helpful" but is probably also arguable: the automatic detection and application of index.html/.wiki/.md files when browsing to a doc directory. It's not a must-have, but would be interesting.

It might seem that one could use /doc/ckout for all of this, with the document repo checked out under the source repo, but that wouldn't work properly on a non-local/public-facing server, as /ckout requires having a checkout and a fossil CGI script is coupled directly with a repo db file, not a checkout.

(3) By Stephan Beal (stephan) on 2019-12-13 23:45:30 in reply to 1 [link] [source]

i feel compelled to demo this because its implications are still kind of blowing my mind. It wasn't clear to me when i first starting tinkering with the /ext feature, but it allows us to effective seamlessly integrate the UIs for multiple repositories for certain purposes.

In this case i have a main source repo:

https://fossil.wanderinghorse.net/r/cwal

It has about 200 pages of accumulated Google Doc-format docs, but GDocs has always had a bad habit of randomly breaking intra-doc links, yet provides no tools to track down broken links and fix them (which is unmanageable in such a pile of documentation), so... at wits end, i've started experimenting with the long process of migrating the documentation to Fossil-side embedded docs.

Sidebar: GDocs can export OpenDocument format, LibreOffice can export that to DocBook, and pandoc can convert that to markdown. The end results are pretty horrid on the main docs (and even worse when converting them to LaTeX), but they provide a usable starting point for the smaller docs which have, so far, been ported. The main doc (just over 160 pages), based on experimentation so far, is going to be a whole other league of Grief to get ported.

Rather by accident, i discovered that the /ext feature can render certain types of static content (wiki, markdown, plain text, images), so started using that as my markdown renderer for porting the docs. It soon became clear that /ext would allow me to host a completely separate doc-only repository embedded into my main repo.

So i set up a separate repo containing of my doc files and ported over some of the smaller docs this evening:

https://fossil.wanderinghorse.net/r/cwal/ext/doc/s2/mod/

The interesting things going on there are:

That's embedded under the main repo using the /ext feature. The embedded doc repo is a checked-out copy, and /ext is serving those files as static content. The /ext feature takes care of embedding those files into the main site's skin. (When put that way, it sounds a bit creepy, but it's genuinely completely non-creepy.) This is distinctly different from embedded docs, which are served directly from a repo db and are thus always current vis-a-vis the latest checkin, whereas static files need an "update" in their checkout dir to get The Latest Stuff.
Links in those markdown files are written as if those documents were embedded in the main repo. e.g., a link from a doc to /finfo/s2/s2.h works even though the doc repo doesn't have s2/s2.h. As of this writing, the first 4 of the embedded doc links from the page linked to above are also part of that nested repo, and they all demonstrate "transparent linking" back into the main repo.
The main repo could, if it wanted to, just as transparently link back to the other repo using docs with links in the form /ext/.... (That's unnecessary in my case, but it's perfectly feasible.)

(Note that this setup requires a current-as-of-just-a-few-hours-ago version of fossil in order to be able to browse index.* files via URIs in the form /ext/some/dir/. Long Live Open Source.)

The Apache-hosted CGI script for the main repo is trivial:

#!/home/me/bin/fossil
repository: /home/me/fossil/cwal.fsl
extroot: /home/me/fossil/ext/cwal

And the doc-only repo is checked out under that extroot (outside of the webroot served by Apache). The dir structure of the doc-only repo mirrors that of the main source repo, so once the migration is done (which i'm now committed to), those files can optionally be moved into the main repo without breaking a single link doc-side.

Anyway... i'm pretty blown away by the possibilities of the /ext feature, and am literally just scratching the surface here.

In closing: @Richard: if this particular aspect of /ext was part of your plan all along, your foresight continues to amaze me. If it wasn't, then congratulations on building a feature which does something incredibly useful, even though that particular something wasn't part of the plan.