Fossil

Timeline
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

200 most recent check-ins by user wyoung

2023-03-28
00:02
Post-sleep edit pass on the new material in §3.2 of the containers doc. ... (Leaf check-in: d21fb267 user: wyoung tags: trunk)
2023-03-27
07:24
Refined the Tcl and Python examples in the new §3.2 of the container doc. ... (check-in: 9baa4423 user: wyoung tags: trunk)
06:33
Switched to a split ENTRYPOINT/CMD scheme for launching the Fossil server in the container. The immediate need is so we override lower-level ENTRYPOINTs provided by mix-in layers, but it's more correct generally. ENTRYPOINT says this is the hard-coded purpose of the container, and CMD gives the arguments to that command. The split is therefore between the mandatory parts and the parts the user might want to override without needing to write their own Dockerfile. ... (check-in: deb99e22 user: wyoung tags: trunk)
04:59
The container now uses BusyBox only in the build and setup stages, leaving just the static Fossil binary in the final stage, plus absolute necessities like a /tmp directory.

This removes the justification for the custom BusyBox configuration, which then means we can use Alpine's busybox-static package in the second stage, saving a bunch of network I/O and build time.

That in turn means we no longer have any justification for jailing the Fossil binary, since there's nothing extra left inside the container for it to play with. Doing this required bumping the Dockerfile syntax back up from 1.0 to 1.3 to get the "COPY --chmod" feature; tested it in Podman, which has had it for two years now.

Doing all of this simplifies the Dockerfile and its documentation considerably. As a bonus, it builds quicker, and it's nearly a meg lighter in compressed image form. Especially for the case of using the container as a static "fossil" binary builder, this is nothing but win. ... (check-in: 79ac06a5 user: wyoung tags: trunk)

2023-03-24
10:21
Comment and whitespace tweaks ... (check-in: 81c30ab9 user: wyoung tags: trunk)
08:27
Switched from a Dockerfile "ADD" command to wget for the BusyBox source tarball because, surprisingly, BuildKit pulls the URL unconditionally under the logic that it can't know whether to cache the pulled data until it has a copy to compare against! This not only means you pull the BusyBox source tarball for each container build even though it's tagged and thus cannot possibly change, it puts a load on GitHub which then causes it to begin throttling each pull, making your local builds slower and slower when iterating on a change set, as in the prior set of commits. By pushing the URL down into a wget command, we cause BuildKit to see an unchanging shell script line (assuming $BBXURL keeps its default) so it *does* cache the pulled layer. ... (check-in: ac955594 user: wyoung tags: trunk)
08:13
Another fixup to the nojail patch to track the previous. (Can't reliably create these patches without having a committed version to diff against, alas.) ... (check-in: c9e4b3d2 user: wyoung tags: trunk)
08:07
Dropped our canned /etc/os-release file entirely, recommending instead that those who need a VM-like container image switch the second stage from "scratch" to one of Google's "distroless" images, which provide that and more. That in turn gets rid of the need for the dummied up /usr/bin and /run, which simplifies the mainstream case. ... (check-in: d778a023 user: wyoung tags: trunk)
07:43
Updated the nojail patch so it applies cleanly atop all these recent Dockerfile changes. No functional change; merely tracks changes in the context parts of the diff. ... (check-in: 2bdd5819 user: wyoung tags: trunk)
07:03
Tiny clarity tweaks to the Dockerfile. No functional change. ... (check-in: 591e3eb9 user: wyoung tags: trunk)
05:23
Removed a reference to /etc/os-release from stage 2 of the Dockerfile. Commit [4cb5c03e] took care of stage 1 only. ... (check-in: 4b41a7f8 user: wyoung tags: trunk)
05:20
Switched from "adduser" and "addgroup" commands for setting up the "fossil" user to direct echo-into-output, same as we already do for the root user. We had to to it for root since the BusyBox implementation of adduser/addgroup won't create these files if they're missing, but that meant we had two different ways of creating users and groups. This not only removes a weak dependency, it's more consistent. ... (check-in: fff11fc6 user: wyoung tags: trunk)
05:17
Added the interactive debugging shell command to the Quick Start section of the containers doc for easy cut-and-paste. ... (check-in: 2f014407 user: wyoung tags: trunk)
2023-03-23
18:03
URL and whitespace fixes to previous. ... (check-in: 9e73519c user: wyoung tags: trunk)
16:40
The /etc/os-release workaround for nspawn's pickiness has caused the feature to go into negative ROI territory. Ripped it out of the mainstream process and made it a manual step for those who need it, in the hopes that this will cause fewer ongoing problems than leaving it as it is. ... (check-in: 4cb5c03e user: wyoung tags: trunk)
15:51
Dropped declaration of Dockerfile syntax version from 1.4 to 1.0. Put it at 1.4 when we were using heredocs, a feature that went from experimental to stable at that version, then failed to drop it back when we replaced the use of heredocs with externally generated files to regain Podman compatibility. ... (check-in: 5b62bfe1 user: wyoung tags: trunk)
15:42
Linked to the Dockerfile from the top of the containers doc. ... (check-in: 2210c15d user: wyoung tags: trunk)
2023-03-13
20:58
Renamed the new "Capabilities" glossary entry to "Capability" since we shouldn't be using a plural top-word entry even though they're defined, transported, stored, and otherwise treated as a group. Also replaced a use of this word in its own definition. ... (check-in: d3f45814 user: wyoung tags: trunk)
20:44
Moved the "snapshot" term in the glossary down into a footnote because it's got assorted problems, making it a much worse overall synonym for "version" even than "UUID". ... (check-in: 733ef88a user: wyoung tags: trunk)
19:54
Expanded the "version/revision/UUID/snapshot" discussion in the glossary into a separate term since these aren't strict synonyms for "check-in", the definition which previously hosted this topic. ... (check-in: 58030a78 user: wyoung tags: trunk)
19:00
Added a new glossary item "Capabilities" to introduce the term and distinguish it from "Permissions", and made several changes to the referenced document to reinforce this distinction and explain why we bother to make it. ... (check-in: 23b91f37 user: wyoung tags: trunk)
2023-03-02
17:24
The recommendation to configure Fossil with the --static flag is semi-obsolete, and the following advice to look further down in the same document for the Docker workaround was wholly obsolete since moving all of this into the dedicated containers.md doc. Fixed all this up, and linked to the "why" answers on Stack Overflow about all of this in a few more places. ... (check-in: d282e42c user: wyoung tags: trunk)
2023-02-28
05:41
The /zip and /tarball built-in help now makes clear that the VERSION/ part of the URL is optional to help avoid confusions like we're seeing in the forum post that sparked this sequence of improvements. ... (check-in: 4717db33 user: wyoung tags: trunk)
2023-02-22
00:09
Clarified one of the 2.21 changelog entries. (Started as fixing a typo.) ... (check-in: 86c4e6bf user: wyoung tags: trunk)
2023-02-15
11:26
Reverted half of commit [4ad86dd5]: it incorrectly moved a CSS style instead of copying it to where it also needed to be. The user-visible effect was that centered Pikchrs varied in size according to their size and complexity, which meant that elements that should've been the same size weren't. ... (check-in: 5ad62aba user: wyoung tags: trunk)
05:16
Cleaned up a few inconsistencies in the Pikchrs in the branching doc in an attempt to fix the smaller-and-smaller diagram size problem currently occuring in this doc. ... (check-in: 239fb5b1 user: wyoung tags: trunk)
2023-02-14
20:54
Updated the PBKDF2 recommendations in the backup doc to track recent changes in best practice due to all these GPU computing fleets coming online. Added a few paragraphs explaining the limits to all of this and why we chose the passphrase lengths we did as examples. ... (check-in: 6a3d6fa6 user: wyoung tags: trunk)
2023-01-24
03:59
Wrapping a few calls to vfile_check_signature() from the new local diff code in unprotect/pop call pairs to squish a DB protection error. ... (check-in: 1b3ef05e user: wyoung tags: ui-local-diff)
03:29
Brought the ui-local-diff branch up to date relative to trunk. It isn't a simple merge, primarily due to all the changes to /vdiff and /fdiff made over the past 2 years. It seems to work as well as it originally did, but it isn't ready to merge down to trunk as-is. ... (check-in: 76fa1657 user: wyoung tags: ui-local-diff)
03:18
Replaced a standalone "diffFlags" variable in the /fdiff handler with use of the new DiffConfig.diffFlags member. No functional change, just a code cleanup found while working on another branch. Making it on trunk to keep that branch's diffs minimal. ... (check-in: 65d97f23 user: wyoung tags: trunk)
2023-01-17
20:40
Small fix to the nojail patch; accidentally lost the [80faedbc] change in the shuffle. ... (check-in: 7a6cf9dd user: wyoung tags: trunk)
20:37
Removed the two "mknod" calls from the Dockerfile in the nojail patch used by Podman rootless containers. Not only is the build user not allowed to run mknod in that case, there will be a /dev tree mapped into the container, causing the commands to fail due to these two basic dev nodes preexisting. ... (check-in: d97a8fb1 user: wyoung tags: trunk)
20:35
No longer running "fossil" with a relative path ("bin/fossil") at the end of the Dockerfile, but instead relying on the hard-coded PATH defined a few sections prior. This allows the same command to work for both the rootful and rootless cases since moving the binary into /usr/bin/fossil to placate nspawn. Before, it was /jail/bin vs /bin, so the difference netted out to nothing. ... (check-in: 80faedbc user: wyoung tags: trunk)
20:32
Fixed a copy-paste error in the Podman sections of the container doc: was using "docker" commands instead of "podman" in a few places. That'll work for people who aliased them, but it's confusing. ... (check-in: 6eefa9b0 user: wyoung tags: trunk)
06:29
Removed use of UPX in the container build process. It complicates the build for a tiny gain while breaking ARM builds. We worked around the ARM-on-ARM case earlier, but it also breaks x86 cross-compilation on ARM. Images are already compressed, and while `upx -9` is stronger compression than whatever Docker Engine is using, it's a small advantage. This does mean the static executable isn't compressed any more on x86, but if you want that, you can UPX it afterward. ... (check-in: da545c9e user: wyoung tags: trunk)
05:28
Generating the /etc/os-release file for the OCI container using autosetup at configure time rather than from a build arg in the Dockerfile at image creation time. This lets us back out the use of heredocs in the Dockerfile, which isn't supported in Podman at all as of this writing and under Docker requires use of BuildKit rather than the legacy "docker build" mechanism.

The primary consequence of doing it this way is that the Fossil version number in that generated file becomes the configure-time version, unconditionally. The old way let you override the FSLVER variable at image build time and have that value put into the os-release file. Under this new scheme, you now have to run "/jail/bin/fossil version" to find out what version of Fossil got baked into the image. ... (check-in: ec8ef573 user: wyoung tags: trunk)

2023-01-13
16:52
Modernized several old fossil-scm.org URLs, changing "http" to "https" where absolute URLs are necessary, and using site-relative URLs otherwise. Also found and fixed a reference to fossil-scm.hwaci.com, which doesn't seem to resolve any more. ... (check-in: 143f1db7 user: wyoung tags: trunk)
16:35
Removed pointless "udc=1" parameters from a few Fossil file links from the docs. ... (check-in: 40d912ae user: wyoung tags: trunk)
2023-01-03
20:13
Added named anchors to the "Image Format vs Fossil Repo Size" doc so I can refer to one in particular. ... (check-in: 7de2410f user: wyoung tags: trunk)
2022-12-16
21:47
Updated the macOS sidebar in the backups.md doc to cover Ventura. ... (check-in: a55042a0 user: wyoung tags: trunk)
2022-12-06
06:13
Grammar and spelling fix pass on the new nspawn material in the containers doc. ... (check-in: 5405aa57 user: wyoung tags: trunk)
2022-12-03
13:17
Typo fixes ... (check-in: 00e4d91e user: wyoung tags: trunk)
11:55
Assorted prose polishing in the new systemd-container section at the end of the containers doc. ... (check-in: 120a2076 user: wyoung tags: trunk)
11:37
Added a few more "container-*" targets to the main makefile to simplify the examples in the containers doc and make the resulting images and containers easier to manage. ... (check-in: b7edb5f1 user: wyoung tags: trunk)
2022-12-02
23:14
Merged two redundant discussions of the consequences of disabling private network virtualization under systemd-container infrastructure, then added better reasons why the reader might care. ... (check-in: 70554336 user: wyoung tags: trunk)
2022-12-01
00:14
Updates to the systemd service doc, primarily to refer the reader to the new containerized runner methods, but also to add other tips. ... (check-in: ad09d3ee user: wyoung tags: trunk)
2022-11-30
23:32
Updated the nojail patch so it'll apply atop the new Dockerfile changes. ... (check-in: 45e0475c user: wyoung tags: trunk)
23:09
Worked out how to get systemd-container (a.k.a. nspawn + machinectl) working with the stock Fossil container. Following the above commits, it's pure documentation. Removed the runc and crun docs at the same time since this is as small as crun while being more functional; there's zero reaon to push through all the additional complexity of those even lower-level tools now that this method is debugged and documented. ... (check-in: 930a655a user: wyoung tags: trunk)
21:27
Added empty /tmp and /run directories to the "OS image" layer of the stock container in case someone is mounting the base layer read-only with tmpfs mounted atop these points. (Seen with "systemd-nspawn --read-only" but might affect other runtimes.) ... (check-in: 0733be50 user: wyoung tags: trunk)
15:23
Container build changes to allow systemd-nspawn to recognize it as an "OS tree:"
  • Added a dummied-up /etc/os-release file
  • Moved several programs from /bin to /usr/bin, since existence of /usr is how it decides if the rootfs you point it at contains an OS image. Bogus, but that's how it is.

Had to switch to buildx to make this work, so I could use heredocs in the first step. ... (check-in: f74ddbce user: wyoung tags: trunk)

14:29
Added "container-clean" target to cleanup after the other container-* targets. ... (check-in: e119d598 user: wyoung tags: trunk)
12:59
Tried to get "--with-tcl=1" working in the containerized build, but failed, so I documented the reason why it isn't going to work given our current design goals and pointed at an alternative with different tradeoffs. ... (check-in: fb1bfce1 user: wyoung tags: trunk)
12:32
Added the FSLCFG Dockerfile build arg and showed how to use it in the containers doc, plus other improvements to the doc while in there. ... (check-in: e2277aad user: wyoung tags: trunk)
12:23
Put a "sleep 1" into "make container-run" before the step that shows the container logs to ensure we show everything it says on startup. Added this on seeing just the first line of output due to a race condition, so I missed the generated admin password. ... (check-in: 4429e10f user: wyoung tags: trunk)
12:19
The "container-run" target now runs "container-image" conditionally, building it only if it wasn't created in a prior step. This allows custom image builds followed by a one-command way of running that built image. Without this, the custom image gets stomped on. ... (check-in: a9e862b8 user: wyoung tags: trunk)
11:44
Reverted the build hack to strip out all but the default and darkmode skins in the stock Dockerfile. That was done to cater to a wish for extremely small ARM builds, for fun, not for any practical reason. It conflicts with a key philosophy behind this container project, to create stock Fossil builds by default. "make container-image" should get you a functionally identical binary inside the container as "./configure && make" does outside it. ... (check-in: 3e95d945 user: wyoung tags: trunk)
2022-11-16
20:53
Prefixing each shell script section in the Dockerfile with "set -x" broke the checks to prevent running UPX on ARM builds. You can still get release container builds on ARM by copying this fixed Dockerfile to your release checkout. ... (check-in: b4c3d9a1 user: wyoung tags: trunk)
2022-10-28
19:48
Also documented the new "clone -u -v" feature. ... (check-in: 0d61fd23 user: wyoung tags: trunk)
19:45
Since it seems my clone -u fixes are going to stick, documented them in the changelog. ... (check-in: 02631e35 user: wyoung tags: trunk)
2022-10-27
17:56
The check for whether to continue during sync due to outstanding "uvgimme" requests was being skipped in clone -u mode due to misordered tests at the end of the client side of the sync protocol. ... (check-in: 52648d03 user: wyoung tags: trunk)
17:15
Since "fossil uv sync -v" turns on UV trace mode, made "fossil clone -u -v" enable that mode as well, since otherwise there's no way to get into UV trace mode during clone. (e.g. There is no global "--uvtrace" option.) ... (check-in: cdd58b1f user: wyoung tags: trunk)
16:01
Consolidated two related tests in the sync protocol to avoid re-testing a flag twice and to bring related code closer together. ... (check-in: 6293b282 user: wyoung tags: trunk)
15:13
Corrected a difference in the case of a SQLite table name. The DBMS doesn't care, but it risks missing relevant references to this table when searching with a case-sensitive text editor. ... (check-in: 1b1887cb user: wyoung tags: trunk)
2022-10-26
18:28
Typo fix in the 2.20 changelog ... (check-in: c3012508 user: wyoung tags: trunk)
2022-10-09
01:11
Grammar fix ... (check-in: 658547aa user: wyoung tags: trunk)
2022-10-07
23:28
Assorted fixes and improvements to the ssl.wiki doc ... (check-in: 27458ef7 user: wyoung tags: trunk)
23:06
Updated the debian/nginx.md doc for Ubuntu 22.04. The biggie is simplifying the TLS configuration, since the manual method we used to have no longer seems to be required with current versions of Certbot. ... (check-in: 716ae7c0 user: wyoung tags: trunk)
22:21
Fixed a few references to the obsolete tls-nginx.md doc. (It became part of the overall nginx.md server doc long ago.) ... (check-in: 780b58bc user: wyoung tags: trunk)
22:15
Assorted updates surrounding my fslsrv wrapper:
  • Reflected improvements from the tangentsoft.com version into this simpler alternative. Although we don't generally recommend use of this script any more, preferring systemd to get autostart on boot and autorestart on crash, www/server/any/none.md still refers to this script, and it feels like a regression to remove it. If someone is interested in simple-as-possible SCGI service, fslsrv is a fit companion.
  • Removed direct reference to fslsrv from www/server/debian/service.md since the indirect reference via the SCGI doc suffices.
  • The full-strength nginx doc now refers to both of these fslsrv variants in a handwavy way, since it's outside the scope of that doc to care how you get your background SCGI servers running.
... (check-in: 1cbcb38c user: wyoung tags: trunk)
2022-10-06
15:42
Added hyperlinks to the new changelog entries referencing the files in question. ... (check-in: 2c127ba7 user: wyoung tags: trunk)
02:44
Closing off the containers project: added the doc to the permuted index, noted the changes in the changelog, and removed all the hedging about WAL mode in the doc, having failed to make WAL fail in this scenario. ... (check-in: 92982dc4 user: wyoung tags: trunk)
2022-09-26
22:29
Replaced most of the speculation in the walmode section of the containers doc with a link to the walbanger project, where we'll be answering this question. ... (check-in: 96633067 user: wyoung tags: trunk)
2022-09-07
09:11
Mentioned containerd+nerdctl in place of runc in the containers doc. A tightened-up version of the prior runc and crun sections are now collected below the Podman section. This gives a better flow: each successive option is smaller than the last, excepting only nspawn, which is a bit bigger than crun. (We leave nspawn last because we can't get it to work!) ... (check-in: 457c14a4 user: wyoung tags: trunk)
07:35
Updated the "nojail" patch for our Dockerfile to track the recent changes: rename back from Dockerfile.in and the layer refactoring. It does essentially the same thing as before. ... (check-in: 19abf0ac user: wyoung tags: trunk)
2022-09-05
08:15
Broke the Dockerfile up into more layers to allow better local caching at build time. Further optimized build time by producing the Fossil source tarball from the local repo instead of hitting the home site if you use the container-image target, since we can be reasonably certain you're working from a repo checkout and thus have all the info available here locally already. ... (check-in: 1da464ee user: wyoung tags: trunk)
01:42
Expanded the paragraph on WAL mode interactions in the container doc into a full section, placed higher up, immediately after the first use of Docker's "--volume" flag, to explain why we don't map just the repo DB file, but the whole directory it sits in. Even if we later convince ourselves WAL is safe under this scenario, it'll be conditional at best, so some remnant of this section must remain, no matter which way the experiments go. ... (check-in: 698587d4 user: wyoung tags: trunk)
2022-09-04
23:46
Renamed Dockerfile.in back to Dockerfile so it can be used as-is on non-autosetup systems. Realized that we can pass the Fossil checkin hash prefix in as a build arg instead of regenerating the file on disk from auto.def. If you use the Dockerfile as-shipped, you get a "trunk" build, which risks a stale cache — it thinks it already has a tarball by that name and helpfully refuses to pull it again — but at least Windows users get *something* without hand-hacking the file. ... (check-in: b0c9c26a user: wyoung tags: trunk)
22:20
Added a /jail/log directory to the container so someone can pass --errorlog and such to the Fossil instance and have a place to put it. It also acts as a mountpoint for appending to a log out on the host. ... (check-in: ed50ceee user: wyoung tags: trunk)
21:58
/dev permissions were too tight in the container. They're still tighter than on a stock Ubuntu box, but they should suffice for Fossil's needs. ... (check-in: 8eeb95e1 user: wyoung tags: trunk)
12:55
Restricted the container listeners to localhost in section 6 of the containers doc, and mentioned a few other items related to reverse proxying with nginx. ... (check-in: c9ab736f user: wyoung tags: trunk)
12:14
Folded info from an exchange with the Podman devs into the container doc. ... (check-in: 80f4a1dd user: wyoung tags: trunk)
11:36
Added section numbers to the containers doc (it was getting confusing) and added a few internal fragment IDs. ... (check-in: 4d51d524 user: wyoung tags: trunk)
11:26
Finished all the new topics planned for the new containers doc, adding sections on rootful Podman containers and on building via Docker but running via Podman, using Docker Hub as an intermediary to avoid building on the remote host. ... (check-in: 9c96e499 user: wyoung tags: trunk)
09:27
Sanitized a local port number out of previous ... (check-in: 3dfa4581 user: wyoung tags: trunk)
09:25
Added my sad tale of failure and woe with systemd-nspawn to the container docs, both as a warning to those who follow, and as a cry for help to someone who can make this work. I can't be bothered to spend more time on it, but there's no point throwing the work away. ... (check-in: 1e8c6655 user: wyoung tags: trunk)
09:15
Documented another cause to modify the "m" variable in the runc examples in the container docs. ... (check-in: bf503088 user: wyoung tags: trunk)
08:09
Added more jq filters to the runc examples to remove further problematic things left in the automatic conversion from the Docker container configuration file to the one we provide to runc. ... (check-in: 4e8c7479 user: wyoung tags: trunk)
07:15
Worked through some difficulties here in applying the runc method on remote systems, then documented what I learned in the containers doc. ... (check-in: 56f4e2ce user: wyoung tags: trunk)
06:28
Small fix to previous ... (check-in: d5695c8e user: wyoung tags: trunk)
06:02
Expanded the runc section of the container doc to cover "bundle" terminology and to show a method for rsyncing the bundle across to a remote host. Also explained why this is a bad idea unless you've got a rather constrained use case, lest people avoid using podman/docker in places where they could provide real value. ... (check-in: f9f13ce7 user: wyoung tags: trunk)
04:32
Documented the runc and crun options for running a container, including the cryptic method for exporting an OCI bundle from Docker, allowing you to use both together: Docker Desktop on your big dev box in the office, then one of the two lightweight runtimes out in the cloud. ... (check-in: c9431ef4 user: wyoung tags: trunk)
02:35
Added explicit instructions for patching the Dockerfile for the nojail/podman method and for mapping a single Fossil repo into the container rather than a directory. Also included my best current advice on using WAL mode in these contexts. ... (check-in: 87a23d2a user: wyoung tags: trunk)
02:15
Removed a TODO-based section of the new containers doc that wasn't meant to be checked in yet. Made a few improvements to the new Podman material as well. ... (check-in: 5adf6c40 user: wyoung tags: trunk)
02:01
Added the "Lightweight Alternatives to Docker" section to the new containers doc, currently limited to a tutorial on converting the stock Dockerfile to work under Podman in its default mode, creating a rootless container. This brings in the second container-related file at the root of the repo, the patch file for this, so we don't have to maintain two nearly-parallel Dockerfiles. As a bonus, it allows us to point to the patch from the prose, making explicit what we had to change. ... (check-in: f0399ea9 user: wyoung tags: trunk)
01:53
Moved the busybox-config file from tools/ into a new containers/ subdirectory. We were using that as a junk-drawer directory, for lack of a better place to put it. Now that we're about to have a second container-related file in the repo, that weak excuse is wearing thin. ... (check-in: b08e2bb7 user: wyoung tags: trunk)
01:39
Referencing the new containers.md file from Dockerfile.in so we can remove a big redundant block comment from it. While in there, made a few style tweaks that will help the ongoing container document expansion. ... (check-in: be8f721d user: wyoung tags: trunk)
2022-09-03
23:34
Extracted the Docker containers material from www/build.wiki and moved it into a new document dedicated to the topic, containers.md. It was already pushing the bounds of how much info we want to provide in a single section of that doc, and it's about to get bigger.

As part of the conversion from wiki format to Markdown, did another edit pass on the doc, improving a few things along the way.

Dropped the "docker-" prefix from all internal IDs, as we no longer need them to disambiguate references to other parts of the build doc. ... (check-in: 7129dc98 user: wyoung tags: trunk)

2022-08-30
01:49
Embroidered the "make container-run" target to make it more convenient. ... (check-in: bc09e28a user: wyoung tags: trunk)
2022-08-29
18:21
The container doc bit on raw sockets now covers the other three Busybox utilities we left out previously. Today's removal of ping and traceroute merely completes the set; it wasn't complete in itself. ... (check-in: b429bd71 user: wyoung tags: trunk)
18:07
Clarified the points in §5.2.1 of the Docker container build doc regarding the reason why the server parent process runs as root. ... (check-in: c2eaa60d user: wyoung tags: trunk)
17:54
Researched, tested, and documented the set of "docker create --cap-drop" options we can add to strip away unnecessary root privileges inside the container without harming normal operation. Belt-and-suspenders: if any bad actor ever got into the container with root privileges, this would help prevent them from affecting anything outside the container. Added that set to the "make container-run" target so they get applied by default in the easy case. ... (check-in: f715add9 user: wyoung tags: trunk)
17:32
Removed ping and traceroute commands from the Docker container. They require raw sockets support, which means if anyone broke into the container and managed a root privilege escalation, they could do a wide array of bad things on any network the container is bound to. ... (check-in: f00a88f8 user: wyoung tags: trunk)
16:01
Polishing pass on §5.2 of the container build doc, "Why Chroot?" ... (check-in: e9860314 user: wyoung tags: trunk)
2022-08-28
17:58
Clarified the parent process user ID vs the child process in the explanation of how the chroot feature interacts with the custom user feature of the Docker container. ... (check-in: f9ddd38e user: wyoung tags: trunk)
17:52
Made a better distinction between bind mounts and Docker volumes in the new Docker section of the build doc. ... (check-in: 958a6af9 user: wyoung tags: trunk)
2022-08-17
05:30
Removed a digression in the gitusers doc about Fossil's new clone-and-open mechanisms. That got moved to the ckout-workflows doc quite some time back, and we already point to it from that same section. There's no reason for the redundancy. Also cleaned up some grammar and typos while in there. ... (check-in: f43eaf01 user: wyoung tags: trunk)
2022-08-16
11:05
Changed the "fossil server --user" flag's argument back to "admin" from "fossil" for the container: I was confusing the Unix user name with the default Fossil repo user name. The new "adduser fossil" stuff doesn't help here; we still want it to be called "admin". ... (check-in: 72d820f3 user: wyoung tags: trunk)
11:04
ARM build fixes for the container:
  • QEMU couldn't cope with "make -j" on the BusyBox step (too many processes) so I changed it to -j11
  • Made the new executable compression step conditional, since there is no upx package in Alpine for either ARM flavor. There's a long bug thread for it on GitHub, which doesn't look to be getting resolved any time soon.
... (check-in: 8849abb7 user: wyoung tags: trunk)
09:39
Minor fixes to the Docker container build process ... (check-in: 454397b0 user: wyoung tags: trunk)
07:14
URL fix necessitated from the Dockerfile.in rename ... (check-in: 2f67bf94 user: wyoung tags: trunk)
07:03
Carved the Docker container image size down still further by stripping out all but two of the stock skins (d* so we get default and darkmode) and packing Fossil and BusyBox with UPX. ... (check-in: e20d044c user: wyoung tags: trunk)
06:49
Fixed an Obi Wan error in the new Fossil version prefix stuff in auto.def: it was extracting the first 13 characters of the hash, not the first 12. ... (check-in: 7ecd23e0 user: wyoung tags: trunk)
2022-08-15
23:21
Added the container-image and container-run top-level build targets to manage dependencies better and to auto-version the build products. ... (check-in: 67386c75 user: wyoung tags: trunk)
23:07
Put the "--user fossil" bit back into the fossil server command for the container. Just ran into a situations where it's still needed. ... (check-in: 4c8cc804 user: wyoung tags: trunk)
22:13
Polishing pass on the container repo storage section of the build docs. ... (check-in: 3e332637 user: wyoung tags: trunk)
22:02
Changed several of the Docker environment variables to build arguments so the user an override them at build time rather than container creation time, and documented them in build.wiki. Using this new mechanism to pull the Fossil source tarball in such a way that we can use the Docker artifact cache without getting stale builds. You can now pass one of the new build args to force the old behavior if you want it. This required generating Dockerfile from Dockerfile.in at configure time, to inject the current Fossil checkin ID. (This busts the Docker cache when the source tree changes.) ... (check-in: f9384383 user: wyoung tags: trunk)
15:32
Adding the BusyBox tarball to the container image with an ADD command rather than wget to avoid triggering GitHub throttling. Unlike the Fossil repo URL, it has a version number baked into it, so it's safe to give it over to Docker's caching behavior. ... (check-in: d06d7c46 user: wyoung tags: trunk)
14:48
Noted the container size shrinkage in the fossil-v-git doc ... (check-in: f21de33e user: wyoung tags: trunk)
14:42
The container now builds Busybox from source so we can remove utilities that are unhelpful inside the container. We leave a lot behind for expansion (e.g. the runit init system, crond, inetd…) but we remove things that have no possible justification, such as modprobe. We remove everything from /bin that's a shell builtin (echo, printf, test…) and we replace a few BusyBox commands (sha[13]sum) with wrapper shell scripts that call Fossil builtins. We cap that off by adding a "sqlite3" wrapper that calls "fossil sqlite3 --no-repository", just for fun. All together, this trims about a meg of fat. ... (check-in: 953f367e user: wyoung tags: trunk)
2022-08-14
19:53
The chown -R bit added to the Dockerfile touches /jail/bin/fossil, which causes "docker build" to promote it back into a new layer, nearly doubling the container size. Doing a chown now only on two directories, restoring it to its sub-9M size. ... (check-in: 00cc9c3e user: wyoung tags: trunk)
19:42
Fossil's chroot feature drops root permissions based on file ownership, but since the container was built with everything-root, its HTTP hit handling children would run as whatever host-side UID/GID pair you used for file ownership. What happened next was complex.

If you let the container create the repo internally, it would be owned as root, so it would drop root permissions for…root! This isn't super-bad, since Fossil is presumed secure and is double-jailed besides. The risk is, if anyone works out an RCE for Fossil, they might be able to get it to create raw sockets or do various other types of escapes despite the double-jail dance.

Attaching a Docker volume brings external permisssions into the container. We were recommending a "chown 0" command on the shared volume to make it similar to the in-container case, but that opens you to the same risks above. If you ignored this and used host-side UID/GID pairs, Fossil would then be left running under IDs that didn't exist internally, which could cause assorted weirdness.

We're now creating an explicit "fossil" user/group pair inside the container and recommending that Docker volumes use these IDs for copied-in files to batten down something that shouldn't've been left flapping.

Updated build.wiki to cover all this. ... (check-in: ba21bc0b user: wyoung tags: trunk)

18:48
Moved the SIGTERM handler up before the "fossil server" HTTP hit handler. We had it clustered with the other signal() calls, but those are to handle signals intended to occur only during CGI processing. This one will normally occur while we're blocked, waiting for the HTTP hit to occur, so it had no useful effect where it was. ... (check-in: d3c55fe0 user: wyoung tags: trunk)
18:01
Changed previous to call fossil_exit() instead of exit(3) so we close our databases before dying. ... (check-in: 7c857d22 user: wyoung tags: trunk)
17:59
The parent process now handles SIGTERM with an explicit exit(3) call when its PID is 1, as when it's running as "fossil server" in a Docker container. Without this, the container host's shutdown process takes a long time because it's waiting on PID 1 to die and eventually has to time out and kill it. ... (check-in: 1d09e607 user: wyoung tags: trunk)
16:19
Markup fix ... (check-in: cf149787 user: wyoung tags: trunk)
16:18
Clarified the fact that the "docker cp" command is changing the name of the repository DB file. ... (check-in: f0b15a37 user: wyoung tags: trunk)
16:15
Slight emphasis fix in previous ... (check-in: 1441c2e6 user: wyoung tags: trunk)
16:13
Edit pass on §5.1 of build.wiki, fixing a number of unclear bits, particularly with regard to images vs containers. ... (check-in: e2b9114b user: wyoung tags: trunk)
2022-08-13
23:39
Using the preceding --chroot fixes to make the Docker container serve the repo from /jail/museum/repo.fossil rather than from the chroot dir, /jail. This then allows us to mount a Docker volume at /jail/museum, which has an independent persistence from the container proper, so we can now rebuild the container without destroying the presumably precious repo. Updated build.wiki to track this change and document the lessons gleaned from doing all of this. ... (check-in: f76e762f user: wyoung tags: trunk)
22:15
Moved the chdir() call within enter_chroot_jail() down below the new repo name canonicalization code to allow use of relative path names. Before, you had to give an absolute path to the repo, since we'd cd'd away from that directory before we started to validate the path. ... (check-in: e9462118 user: wyoung tags: trunk)
22:14
Moved the setting of g.fJail flag into the repo = "/" case since it exists only to communicate the chroot status to --repolist mode. (This confirms the speculation in the prior commit's comment: the prior behavior existed to serve repolist mode only.) ... (check-in: 324d232c user: wyoung tags: trunk)
21:21
Fixed the --chroot flag to "fossil server" and "fossil http" to allow it to work in conjunction with the single-repository case. Before, it blindly assumed --repolist mode. ... (check-in: 6f92ad99 user: wyoung tags: trunk)
2022-08-12
17:01
Fixed pointless use of interwiki link in the new section 2.2 material of fossil-v-git. ... (check-in: 73c95307 user: wyoung tags: trunk)
2022-08-06
22:13
Fixed a few stray parens in the new material in the fossil-v-git doc, left behind from a prior edit. ... (check-in: ea13701c user: wyoung tags: trunk)
22:08
Typo fix ... (check-in: b628a883 user: wyoung tags: trunk)
20:30
Fixed a problem in image naming in the new Docker container doc in build.wiki reported on the forum. ... (check-in: 509447a2 user: wyoung tags: trunk)
19:56
Did away with the temporary src.tar.gz file in the new Docker container by streaming the output of wget straight into tar's stdin. This cuts the build time by about five seconds, presumably due to the saving from unnecessary file I/O. Also replaced the explicit "cd src" afterward with an out-of-tree build configuration, since it doesn't matter if we clutter the first stage's /tmp dir. ... (check-in: 289c9b50 user: wyoung tags: trunk)
19:34
The build docs for "./configure --static" now reference the section further down on Docker, since you may need to use this indirection to get --static to produce something suitable. ... (check-in: 7bfd7413 user: wyoung tags: trunk)
04:24
Replaced Jan Nijtman's Dockerfile with a new one that does a 2-stage build. The first stage runs atop Alpine Linux instead of Fedora, reducing the initial build from ~635 MiB to about 16.

Rather than stop there, I then made it multi-stage, copying two key static binaries — Fossil and Busybox — over from the first stage into a fresh-from-scratch container and set it up to run the former jailed away from the latter.

The result is under 9 MiB, and it's as secure as one can hope, given that it starts up in "PUBLIC" mode. The new build doesn't have all the extra features turned on that the old one did, but it seems right to build the container with Fossil in its default configuration. If you want something else, copy the Dockerfile, hack it, and make it do what you want instead.

Having done all this, I replaced the one-off Dockerfile inline in section 5.0 of the build doc with a reference to this new Dockerfile and rewrote the section in terms of the new capabilities.

Finally, this lets us brag on how small the container can be, as compared to the Gitlab-CE container. Before, we were comparing a standalone binary to the container, which wan't entirely fair. (The desire to produce such a container was the spark that kicked this project off.) ... (check-in: 77d603c6 user: wyoung tags: trunk)

2022-08-05
12:05
Assorted improvements to the first few sections of the fossil-v-git doc, mainly in updating them to track changes to world facts and to clarify the presentation. ... (check-in: c7afd68b user: wyoung tags: trunk)
2022-07-18
01:39
Still moer grammar fixes in fossil-v-git ... (check-in: e28c25e4 user: wyoung tags: trunk)
01:08
More grammar fixes ... (check-in: 9f135f2f user: wyoung tags: trunk)
01:07
Grammar fix to the fossil-v-git doc ... (check-in: f36fb951 user: wyoung tags: trunk)
2022-07-17
23:48
Changed a number of "a" articles followed by vowels in docs and comments to "an", per a forum post. ... (check-in: 99a319bd user: wyoung tags: trunk)
23:23
Mentioned "fnc stash" at the end of the section of gitusers where it talks about alternatives to "git add -p" and such. ... (check-in: b3b2c1ab user: wyoung tags: trunk)
2022-07-11
19:49
Continued the edit pass on the main body of the gitusers doc, shy of the case studies, mainly doing minor style tweaks. Biggest substantial change is to rewrite the colorized diff section to cover the changes in Fossil 2.17, and to present the alternatives in a more logical order. ... (check-in: c026fb9a user: wyoung tags: trunk)
18:50
Fix to the fix. :( ... (check-in: e3f9584e user: wyoung tags: trunk)
18:50
Fixed a broken internal link in the gitusers doc resulting from moving the museum tree pikchr into the glossary. ... (check-in: 116d8c75 user: wyoung tags: trunk)
2022-04-20
14:59
Rewrote the login-groups doc, making it both more clear and more detailed. This started out as clarifying a confusion brought up on the forum, but experimentation kept bringing up new and interesting restrictions and interactions that I felt were worth documenting. ... (check-in: 697cf6fb user: wyoung tags: trunk)
2022-04-11
09:33
Brought the "Fossil grep vs POSIX grep" doc up to date relative to the merged grep-enhancements branch. ... (check-in: caba4b01 user: wyoung tags: trunk)
2022-04-10
18:56
Added the "--page wcontent" bit to the new wiki versioning example in the glossary. ... (check-in: b05a07a9 user: wyoung tags: trunk)
18:53
Grammar tweaks to previous ... (check-in: ccd5cacc user: wyoung tags: trunk)
18:49
Added "Embedded Documentation" section to the glossary per larrybr's request in /chat. It's a Fossil-specific term of art. It also gives us a place to contrast with the wiki; that could live in the docs for the wiki or embedded docs instead, but the glossary is where newbies go to get oriented on terms, so the "but which should I use" question falls right out of the terminology. ... (check-in: e583b48a user: wyoung tags: trunk)
2022-03-28
08:34
Updated a reference to macOS 11 from the backup doc: the condition it warns against is still true as of macOS 12.3. ... (check-in: 1bb4147f user: wyoung tags: trunk)
2022-02-28
23:32
Calling db_open() to determine if a given repository is valid rather than a hand-rolled sqlite3_open() call. This then allows us to call db_looks_like_a_repository() to determine if the DB is a valid repo rather than duplicate the checks it already has in another nearby context. This is part of the apndvfs vs normal-case stuff done in prior commits, consolidating the notion of "valid" to a single spot in the code. ... (check-in: 69145d9d user: wyoung tags: trunk)
20:35
Simplified an overly-clever test for a file size being an even multiple of 512 bytes. Compiler Explorer says GCC 11 generates the same code both ways, at least, and it isn't in a CPU-critical code path anyway. Also added a comment referring to this new, simplified code, to prevent a recurrence of the problem fixed by the prior commit. ... (check-in: c67d5401 user: wyoung tags: trunk)
19:23
Reverted a check for the repository size being an even multiple of 512 bytes as a test for validity. Introduced in an omnibus commit for obscure reasons, it causes some valid clone operations to fail, as originally reported on the forum. ... (check-in: 4a2d0e78 user: wyoung tags: trunk)
2022-01-16
18:42
The output of "fossil configuration --help" had two different ways -R was described, one wrong. The newer one was of a more consistent format with the rest of the help ([decd537016 | thus why it was added]) so removed the older one and reworked the newer one to be more accurate. ... (check-in: 6cb0fc25 user: wyoung tags: trunk)
2022-01-08
04:58
Moved the GitHub forking stats down in the fossil-v-git doc to a spot where it fits better. Updated the stats, and tightened up the prose. ... (check-in: c0269e34 user: wyoung tags: trunk)
04:19
Fixed a Markdown-ism in a Fossil wiki doc. ... (check-in: ab48b9da user: wyoung tags: trunk)
04:19
Made the "scale" issue in fossil-v-git less of a false dichotomy. ... (check-in: 5171e591 user: wyoung tags: trunk)
2021-12-22
09:18
Added more reasons not to use Fossil as a whole-system configuration backup utility in the glossary point about not using Fossil to store files scattered hither-and-yon over a filesystem. ... (check-in: 79948097 user: wyoung tags: trunk)
09:14
A few clarifications to the new glossary. ... (check-in: 78aa4394 user: wyoung tags: trunk)
2021-12-12
16:18
The "Summary Line Convention In Commit Comments" section in gitusers.md now covers the related setting under Admin -> Timeline. ... (check-in: c33ffed3 user: wyoung tags: trunk)
15:41
Extracted the glossary to a new document from the old "why use Fossil" doc, expanded it considerably, converted it from Wiki to Markdown, and updated the links to point to its new location. ... (check-in: a58d952f user: wyoung tags: trunk)
2021-10-07
13:01
Fixed a copy/paste error in www/sync.wiki per an anonymous forum post. ... (check-in: 3f736de9 user: wyoung tags: trunk)
2021-09-21
16:42
Fixed a paren nesting bug that prevented multi-character hashtags from being processed. It would stop at the second character. ... (check-in: e211f1ab user: wyoung tags: markdown-tagrefs)
16:38
Added '@' and '#' prefixes in spans. Initial commit ate them. ... (check-in: 398cfa0b user: wyoung tags: markdown-tagrefs)
16:19
Initial implementation of "span data-foo" wrappers around @name and #tag references. Seems functional on a test-markdown-render basis, but the definitions of what counts as a reference and what to do with them still remains to be handled. ... (check-in: 31a607d3 user: wyoung tags: markdown-tagrefs)
09:47
Fixed a comment to match the code, as reported on the forum. ... (check-in: 898b8f20 user: wyoung tags: trunk)
06:22
Assorted small improvements to the gitusers doc ... (check-in: 7cd51fa3 user: wyoung tags: trunk)
2021-09-19
14:37
Restored "diff --command CMD" flag that got lost in the latest diff refactorings. (One-off method for supplying the diff-command setting.) ... (check-in: cae7036b user: wyoung tags: trunk)
03:55
Fixed a few minor errors in the new http-over-ssh.md doc and added a memorial to a fallen hero. ... (check-in: 9d4a1327 user: wyoung tags: trunk)
03:34
Put a space after all "###" used as inline headers in www/customgraph.md to make them behave the same if we follow the CommonMark spec and require at least one space or tab after the last #. All other www/*.md files already do this. ... (check-in: bc08b097 user: wyoung tags: trunk)
2021-09-18
01:37
Assorted small improvements to the new http-over-ssh.md doc. ... (check-in: d8f47075 user: wyoung tags: trunk)
00:51
Added the www/server/any/http-over-ssh.md doc, an elaboration of Andy Bradford's OpenSSH ForceCommand based solution for forcing access via ssh:// URLs to go through a wrapper script that rewrites the command, exchanging "test-http" for "http", causing the Fossil RBAC system to come into play in a secure fashion. Linking to it from the top-level "server/" index and from the #webonly section of the caps/ index where it cites this limitation. Reworked the latter section now that we have a documented alternative. ... (check-in: be8ed971 user: wyoung tags: trunk)
2021-09-17
21:48
Clarified the read/write access issue in the #webonly section of the main user capabilities doc. ... (check-in: 391bc369 user: wyoung tags: trunk)
02:32
Updated the JS doc's section about the hamburger menu to reflect the recent addition of this menu to other stock skins. ... (check-in: 36d84427 user: wyoung tags: trunk)
02:02
Converted all uses of the obsolete named anchor feature from HTML before 4.0 to use fragment identifiers instead. (www/* subtree only.)

Where possible, changed constructs like

<a name="foo"></a><h3>
to
<h3 id="foo">

Also fixed a few cases where the link target came after a header so the browser would scroll the header off the screen when visiting the targeted section.

Added a 50em pad at the bottom of one such edited doc to allow the intra-doc link targets to be useful since it's a short enough doc that on sufficiently tall browser windows, scrolling isn't possible, so using those anchors has no visible effect. ... (check-in: 93cee1f5 user: wyoung tags: trunk)

2021-09-09
16:19
Edit pass on the blockchain doc: mainly clarity improvements, but also some typo and grammar fixes. ... (check-in: c34ca629 user: wyoung tags: trunk)
2021-09-02
07:56
Half of [d700f5ff4f215c69 | the prior commit] was unnecessary. Backed it out. ... (check-in: 9a4e87a3 user: wyoung tags: trunk)
07:52
Added db_unprotect/pop wrappers around the "clone --ssh-command" handling to avoid a bogus "unauthorized change to protected setting" error, as reported on the forum. ... (check-in: d700f5ff user: wyoung tags: trunk)
2021-08-31
14:13
Removed the link to /setup_smtp from the main /admin page: the backing feature was removed several months ago. Addresses an anonymous forum post. ... (check-in: 7aaee6fc user: wyoung tags: trunk)
14:10
Markdown tweak in www/alerts.md to avoid semi-redundant constructs like "[MTA][mta]". ... (check-in: 897bd90b user: wyoung tags: trunk)
14:03
Fixed several "descendents" typos. (It's "descendants".) There's still one remaining in sqlite3.c, but of course fixing that here won't help. ... (check-in: 85f119dc user: wyoung tags: trunk)
13:44
Small grammar tweaks to previous ... (check-in: c2f0063f user: wyoung tags: trunk)
13:29
Further tweaks to §2.5.1 of the fossil-v-git doc: spelling and grammar fixes, updated references, clarifications... ... (check-in: c00b6a48 user: wyoung tags: trunk)
13:14
Updated the fossil-v-git doc to talk about "fossil patch" in addition to "fossil bundle". ... (check-in: e3e06d31 user: wyoung tags: trunk)
2021-08-19
10:38
Fix for previous. :P ... (check-in: 3befe7aa user: wyoung tags: trunk)
10:37
URL fix addressing a report on the forum. ... (check-in: 0da45710 user: wyoung tags: trunk)
2021-08-11
18:56
Adjusted the optional cscope feature so it collects symbols only from the src subdir, primarily to get the bld/* stuff out of the way. ... (check-in: b6cc6f68 user: wyoung tags: trunk)
2021-08-02
10:09
Typo fix in previous ... (check-in: e3066ede user: wyoung tags: trunk)
08:49
Clarified the double-quoting and {} quoting rules for TH1 in the docs. ... (check-in: a1e41529 user: wyoung tags: trunk)
2021-07-23
21:47
Updated the caps/index.md doc to reflect the recent "sxy" change ... (check-in: 34de6214 user: wyoung tags: trunk)
19:48
Comment typo fix in previous ... (check-in: ec5efceb user: wyoung tags: trunk)
19:35
The default user capability string for the test-http command — used by ssh:// URLs despite the "test-" prefix! — is now "sxy" to grant all permissions as claimed in the docs. While this is objectively correct, it doesn't solve the actual problem I was chasing, being UV sync failure over SSH per a recent forum post. ... (check-in: 129e3958 user: wyoung tags: trunk)
2021-07-20
20:57
Updated the link to cURL's cacert.pem package from the "SSL" doc ... (check-in: af7bbdce user: wyoung tags: trunk)
2021-07-15
20:25
Updated the systemd service article to account for my experience following the instructions on a CentOS 8 box. (It was originally written for an older Debian type platform.) ... (check-in: 74670ab1 user: wyoung tags: trunk)
16:55
Updated the meld example for the gmerge help output, from a fix suggested on the forum. ... (check-in: 375589e5 user: wyoung tags: trunk)
2021-06-22
18:19
Reworked the MinGW outdated instructions in www/build.wiki. It was still warning about a MinGW bug from 2014, and it didn't cover the cross-compilation details at all. That in turn gives us a basis for explaining why cross-compilation may matter even to those who don't realize they're doing so, as in the Cygwin and WSL cases, which then explains why we recommend against using MinGW Make and the USE_WINDOWS=1 mode. ... (check-in: 2c66a539 user: wyoung tags: trunk)