Fossil User Forum

Postmorten of migration from GDocs to embedded docs
Login

Postmorten of migration from GDocs to embedded docs

Postmorten of migration from GDocs to embedded docs

(1) By Stephan Beal (stephan) on 2019-12-19 02:22:34 [link] [source]

Follow-up/postmortem:

As mentioned in /forumpost/7545d7a31e, i've spent much of the past week porting over 200-ish pages from Google Docs to Fossil embedded docs. The migration itself took right at 4 days, followed up by incessant tweaking for tweaking's sake. This post is about what i learned in the process...

  1. This should have been done years ago. Though i've been an active Fossil user since Christmas of 2007, i've never made any significant use of the embedded docs feature (which came a few years later). Despite the convenience of GDocs, having the docs directly next to their sources, and able to link to those sources with just a few bytes of markup, is a huge plus for me.

  2. The /ext feature was a true godsend here. Specifically, it can serve static content and integrate certain file types (e.g. markdown) into the repo's skin. This allowed me to port the docs in a doc-specific repo so that the migration wouldn't clutter up the main repo's log with dozens upon dozens of doc-only commits, and yet still have the docs integrated with the main site. When complete, the contents of the doc-specific repo were moved as-is into the main repo. Because of how the /ext integration works, my inter-doc links did not require any modification and links to in-repo resources, e.g /finfo/foo/bar.c, worked identically under both /ext and after importing the docs into the main repo.

  3. (Continually) Tweaking the skin to get it more or less to my liking was a bit of a drag due to the (admittedly reasonable) hoops imposed by the UI's skin-editing process and the editing limitations of plain textarea elements when working with CSS. It probably would have been much simpler to hack the skin using the --skin flag to fossil ui, or even moving all relevant CSS/JS to files under /ext and emacsing them from there. For some reason, fossil conf push skin sends the skin to my remote/live site but does not activate it, requiring frequent manual activation of the skin modifications into the live site. fossil conf pull skin from the live site to my local one does activate it, though. That seems like a bug. (Just tested it twice: after pushing to the live site, i must manually click on Step 3 in the skin admin page to get it activated. Pulling the skin from the live site, after making changes there, does not require this step. Weird.)

  4. Integrating site-wide syntax highlighting was fairly trivial, as described in /forumpost/9150bc22ca, including creation of a highlighting module specific to the custom scripting language for which the docs were written.

  5. i learned a good deal about Markdown syntax in the process, and am pleasantly surprised at fossil's compatibility/predictability in that regard, even some of the weird stuff like ending a line with a backslash followed by two spaces (which often came in handy in lists). The only feature i missed is ~~strikethrough~~, but markdown allows embedded HTML (which i only just learned this past week), so the <S> tag can do that job: strikethrough.

This migration might (might) represent the single largest collection of Fossil embedded docs outside of the main fossil repo:

https://fossil.wanderinghorse.net/r/cwal/doc/trunk/s2/

[stephan@lapdog:~/fossil/cwal]$ wc $(find . -name '*.md' | grep -v fossil-ext)
 <59 files snipped>
 15761  98614 654263 total
 ^lines ^words ^characters

If there's a bigger one out there, my money says Warren has it!

(2) By Stephan Beal (stephan) on 2019-12-19 02:35:48 in reply to 1 [link] [source]

The follow-up continues:

Rather by accident, i just learned that using a path like this in the site header:

<link rel="stylesheet" href="$baseurl/doc/ckout/doc/highlightjs/hybrid.css" type="text/css" />

(Note the reference to doc/ckout)

... actually works, even if there's no checkout. It apparently takes the trunk or the tip (not sure which) if there is no checkout. That simplifies my configuration by allowing me to always use ckout and get consistent results locally (where there is a checkout) and the remote (where there's not). This whole time i've been carefully using separate headers for the local and remote site. Sigh.

(7) By Stephan Beal (stephan) on 2019-12-19 19:05:10 in reply to 2 [link] [source]

It [doc/ckout] apparently takes the trunk or the tip (not sure which) if there is no checkout

For completeness' sake: it's the tip, as documented in the /doc help.

For those who are wondering, "why tip instead of trunk?": "trunk" is just a naming convention, and repos don't have to have a trunk (it can be removed). The tip, on the other hand, is "the newest version, regardless of which branch", and is something all repos except completely empty ones necessarily have. (Noting that fossil doesn't create completely empty repos of its own volition - that's something users have to take extra steps to do if they want.)

(3) By Chris (crustyoz) on 2019-12-19 12:05:54 in reply to 1 [link] [source]

Well, I can't claim to have a bigger (whatever...) but I've been using embedded docs for a decade. Here are the numbers for my two largest doc only repositories:

chris@wirth:~/xxx/server-docs$ wc $(find . -name '*.md')
<107 files snipped>
 11591  60324 428989 total
 ^lines ^words ^characters

chris@wirth:~/xxx/v3$ wc $(find . -name '*.md')
<43 files snipped>
  6097  27987 197111 total
 ^lines ^words ^characters

These files were written and are maintained in my preferred text editor rather than having migrated from some other text generating program. Most lines are <= 65 characters in length so they still look good in a plain text printout as well as being displayed (mostly) via fossil both locally and from a remote server.

Chris

(4) By Stephan Beal (stephan) on 2019-12-19 12:57:40 in reply to 3 [link] [source]

Well, I can't claim to have a bigger (whatever...) but I've been using embedded docs for a decade.

At 10 years old, yours is certainly... longer... lived ;). And now you've got me wondering how old that feature is...

[stephan@lapdog:~/fossil/fossil/www]$ f finfo embeddeddoc.wiki | tail -1
2008-05-19 [4b826eed14] Add a page describing how to do embedded documentation. (user: drh, artifact: [97f9390860], branch: trunk)

My recollection was (incorrectly) 2010, plus or minus a year.

(5) By Warren Young (wyoung) on 2019-12-19 16:56:32 in reply to 1 [link] [source]

If there's a bigger one out there, my money says Warren has it!

I wrote about 200 kWords of fiction for NaNoWriMo this year, so I made a Fossil repo from it just to fulfill your prediction. :) (I'd been content to keep it only in Calibre before now.)

It's organized as two thin novels — 85 kWords — and one partial third, but taking them as a total, it's roughly in the middle of the range set by Tom Clancy for single novels.

I don't mean to diminish the work done on these repos, but Fossil should support repos holding small libraries worth of book material. Surely someone has bigger prose repos than these?

(6) By Offray (offray) on 2019-12-19 18:56:31 in reply to 5 [link] [source]

I use Fossil mostly to hold documentation (prose) repos, as my source code is mostly done in Pharo and hosted in SmalltalkHub.

I have the Grafoscopio, Dataviz and Brea packages documentation, the Data Journalism Handbook, the port of Data Feminism book, the Documentathon book and even my PhD thesis.

As I was learning to use Fossil as I advanced in the documentation practices, the more recent documentation/repos have better practices, like using unversioned files for storing PDFs and adding manifests and hashes to link PDF and other derived files with the bundle of source files that compose a particular derived file. There is a lot to clean of my rookie practices on many of those repos though.

Just my two cents.

(8) By Peter MacDonald (pmacdona) on 2020-01-21 23:35:10 in reply to 1 [link] [source]

I've been using embedded docs for a few years, and one issue that used to be just a nit, is as of the Github sync feature is now a show-stopper. That is, we can not add our own mimetypes, as they are hard-coded in doc.c aMime[].

ie. this displays the file:

https://github.com/pcmacdon/jsish/blob/master/js-demos/thread.jsi

but this does downloads the file:

https://jsish.org/fossil/jsi3/doc/ckout/js-demos/thread.jsi

Previously, was using the "/file/" prefix to get around this, however this is not an option if we are using git for both, nor is using ?mimetype...

Would be nice if there was a config option for adding your own mimetypes.

(9) By Stephan Beal (stephan) on 2020-01-22 00:50:50 in reply to 8 [link] [source]

we can not add our own mimetypes

That's been a minor nit of mine for years, but i was never motivated to fix it because (A) it's so rare that i need to and (B) i've got commit access to add new ones ;) (like webp, just recently). When idly considering a more widely-usable solution, i've stumbled on "where does it belong?"

It occurs to me now that we might be able to use a new versioned setting for this, with either a mapping of extension-to-mimetype.

Something like:

cat .fossil-settings/mimetypes
web image/webp
foo application/foo
...

Would something like that suffice for you? (Obviously, the pages which use mimetypes would need to be bent to make use of that info, but that doesn't sound so bad.)

(10) By Stephan Beal (stephan) on 2020-01-22 02:22:24 in reply to 9 [link] [source]

@Peter: would this serve your needs?

First, with a non-versioned setting:

$ f help mimetypes
Setting: "mimetypes" (versionable)

A list of file extension-to-mimetype mappings, one per line.

$ f set mimetypes
mimetypes            (local)  foo application/x-foo
bAr application/x-bar

$ f test-mimetype a.foo aaaaa.bar z.baz
a.foo                -> application/x-foo
aaaaa.bar            -> application/x-bar
z.baz                -> application/x-fossil-artifact

Second, with versionable a settings:

$ echo 'baz application/x-baz' > .fossil-settings/mimetypes

$ f test-mimetype a.foo aaaaa.bar z.baz
setting mimetypes has both versioned and non-versioned values:
using versioned value from file "/home/stephan/fossil/fossil/.fossil-settings/mimetypes"
(to silence this warning, either create an empty file named
"/home/stephan/fossil/fossil/.fossil-settings/mimetypes.no-warn" in the
check-out root, or delete the non-versioned setting with
"fossil unset mimetypes")
a.foo                -> application/x-fossil-artifact
aaaaa.bar            -> application/x-fossil-artifact
z.baz                -> application/x-baz

Oh, right...

$ f unset mimetypes
$ f test-mimetype a.foo aaaaa.bar z.baz
a.foo                -> application/x-fossil-artifact
aaaaa.bar            -> application/x-fossil-artifact
z.baz                -> application/x-baz

(11) By Peter MacDonald (pmacdona) on 2020-01-22 03:23:21 in reply to 10 [link] [source]

Sure. All sounds good to me, especially compared to the alternative of modifying doc.c, and rebuilding, which BTW was how I came to report the Makefile.classic bug.

(12) By Peter MacDonald (pmacdona) on 2020-01-22 03:40:53 in reply to 10 [link] [source]

Well, as usual, there is one more thing :)

In a .md doc file on Github a link to another file "../lib/web/jsi.js" shows the file in the same form as when using /file prefix in fossil, eg.

https://jsish.org/fossil/jsi3/file/lib/web/jsi.js

but on fossil it shows as raw and unadorned.

https://jsish.org/fossil/jsi3/doc/ckout/lib/web/jsi.js

I'm not sure how to reconcile this, because the first allows you to download individual files from fossil using "save link as". Whereas with Github that doesn't seem to be possible.

Regardless, it is only markdown links are where this is really useful, from my perspective anyways.

(13) By Stephan Beal (stephan) on 2020-01-22 03:52:35 in reply to 12 [link] [source]

Having doc links pass on non-doc files as-is qualifies as a feature for me. That's one of those "either way is correct (or not wrong)" features where github and fossil (and likely many other md-serving services) take different routes.

(14) By Peter MacDonald (pmacdona) on 2020-01-22 04:10:16 in reply to 13 [link] [source]

I have to agree. In fact, I think a big reason I resisted using github for so long was it forces you to download the whole zip just to get one file.

(15) By Chris Mann (chris2fr) on 2021-07-18 16:20:52 in reply to 1 [link] [source]

I would like Strikethrough with ~~.

I would also like disregarding YAML frontmatter with

/\A---(.|\n)*?---/

(16) By Stephan Beal (stephan) on 2021-07-18 18:16:42 in reply to 15 [source]

I would like Strikethrough with ~~.

Once you get used to the strike tag, the double-tilde approach seems like overkill. <s>strike</s> ==> strike

The main advantages of that tag are that it works in both of fossil's wiki formats as well as standard HTML, so no new syntax is required.

I would also like disregarding YAML frontmatter with

Why would/should we treat embedded yaml specially? If it's used as configuration for a specific markdown formatter than it's extremely unlikely to gain any traction here: fossil's markdowns support is for fossil itself, not arbitrary applications (and fossil doesn't use yaml).

All of that being said: patches often talk louder than objections.