Trying to migrate to HTTPS for our in-house Fossil server, I asked the keeper of our private CA to issue a certificate and private key. Adding these to the Fossil server command line via -cert and -pkey worked wonders, and now our web browsers are able to talk to our Fossil server with confidence. Naturally, that works because corporate IT configured our desktop systems to trust our CA.

However, I'm on my own when it comes to our Linux-based development systems, which will also need to import our CA's certificate. Starting with a Slackware64 virtual machine to develop the process, I put it into /usr/local/share/ca-certificates/XXX.crt and ran update-ca-certificates. That's all it took for programs such as wget to be happy. To double-check, I confirmed update-ca-certificates indeed put my CA's certificate in /etc/ssl/certs/ca-certificates.crt.

Alas, this is not enough for the Fossil client, which cannot verify the certificate received from the Fossil server:

$ fossil clone https://XXX/XXX XXX.fossil
Unable to verify SSL cert from XXX
  subject:   O = XXX, OU = XXX, CN = XXX
  issuer:    DC = XXX, DC = XXX, CN = XXX
  notBefore: 2024-12-05 16:19:31 UTC
  notAfter:  2025-12-05 16:19:31 UTC
  sha256:    XXX
accept this cert and continue (y/N/fingerprint)?

Here's how Fossil is configured:

$ fossil ssl
OpenSSL-version:      OpenSSL 3.4.0 22 Oct 2024  (0x030400000)
Trust store location
  SSL_CERT_FILE:
  SSL_CERT_DIR:
  ssl-ca-location:    /usr/local/share/ca-certificates/XXX.crt
  OpenSSL-cert-file:  /etc/ssl/cert.pem
  OpenSSL-cert-dir:   /etc/ssl/certs
  Trust store used:   /usr/local/share/ca-certificates/XXX.crt
ssl-identity:

I'm anxious about the private CA's certificate itself being listed as the trust store. That should be /etc/ssl/certs/ca-certificates.crt. Overriding with SSL_CERT_FILE doesn't get me any farther, though. I still get identically the same complaint about not being able to verify. All that changes are the SSL_CERT_FILE and Trust store used lines in the output of fossil ssl.

I tried stepping through a debug build of OpenSSL but gave up pretty quickly due to how complicated it is. At least I was able to learn the internal error is X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT.

How to proceed?

What’s the output of

$ fossil set | grep ssl

Properly set up, everything should be blank, because you're using the platform defaults as intended. Here's what I have on an AlmaLinux box here, which works for both local TLS and public:

$ f ssl
OpenSSL-version:      OpenSSL 3.2.2 4 Jun 2024  (0x030200020)
Trust store location
  SSL_CERT_FILE:      
  SSL_CERT_DIR:       
  ssl-ca-location:    
  OpenSSL-cert-file:  /etc/pki/tls/cert.pem
  OpenSSL-cert-dir:   /etc/pki/tls/certs
  Trust store used:   /etc/pki/tls/cert.pem
ssl-identity:        
$ f set |grep ssl
ssl-ca-location     
ssl-identity        

I specified the platform above because each one is free to configure their OpenSSL package's paths differently. I have to do something similar but not identical under the Homebrew distribution of OpenSSL on macOS, for example.

Bottom line, be careful reading guides on the Internet not made for your platform. You'll want to find a Slackware-specific one for best assurance that it actually applies properly.

Thank you for your response. At this instant, it's:

ssl-ca-location      (global) /usr/local/share/ca-certificates/XXX.crt
ssl-identity

That setting is left over from an experiment. Allow me to unset it and try again:

OpenSSL-version:      OpenSSL 3.4.0 22 Oct 2024  (0x030400000)
Trust store location
  SSL_CERT_FILE:
  SSL_CERT_DIR:
  ssl-ca-location:
  OpenSSL-cert-file:  /etc/ssl/cert.pem
  OpenSSL-cert-dir:   /etc/ssl/certs
  Trust store used:   /etc/ssl/certs
ssl-identity:

Much better. That's what I expected to see. Sorry I forgot about my experimental setting, since I did it on an earlier date.

Alas, just like the SSL_CERT_FILE override, this still doesn't get me across the finish line. Fossil still cannot verify the certificate. Let's get strace involved. Here's the complete list of calls to openat():

/etc/ld.so.cache
/lib64/libfuse.so.2
/lib64/libm.so.6
/lib64/libresolv.so.2
/lib64/libz.so.1
/lib64/libc.so.6
/lib64/libdl.so.2
/lib64/libpthread.so.0
/root/.config/fossil.db
/dev/urandom
/root/fossil-test/devops.fossil
/root/fossil-test/devops.fossil-journal
/root/fossil-test
/root/fossil-test/devops.fossil
/root/fossil-test/devops.fossil
/root/fossil-test/devops.fossil-journal
/root/fossil-test
/root/.config/fossil.db
/root/.config/fossil.db-journal
/root/.config
/root/.config/fossil.db-journal
/root/.config
/root/fossil-test/devops.fossil-journal
/etc/ssl/openssl.cnf
/etc/nsswitch.conf
/etc/host.conf
/etc/resolv.conf
/etc/hosts
/etc/ssl/certs/XXXXXXXX.0
/etc/localtime

I verified /etc/ssl/certs/XXXXXXXX.0 is indeed the correct certificate for our local CA, also that the X's are the cert's subject hash, basically the SHA-1 of the DN. This "file" is a symlink to a symlink to the certificate I installed. openssl x509 -noout -hash -in /usr/local/share/ca-certificates/XXX.crt gives this value. Anyway, this proves Fossil and OpenSSL are reading the correct certificate. But for some reason, even though wget is happy, Fossil is not. Unsurprisingly, using strace on wget reveals it reads the same XXXXXXXX.0 file as Fossil, though I see it also tries (and fails) to read /etc/ssl/cert.pem which doesn't exist.

Yes, I'm running as root, since this is in a virtual machine created expressly for building and experimenting with Fossil and other devops tools. No reason to mess with unprivileged users when we're already in a tight sandbox. If you think a regular user might fare better than root, I can give that a try.

Apologies for my redactions, I just don't want to take any chances with people thinking I'm compromising security by divulging hostnames, key fingerprints, or whatever.

There must be some difference in how wget and Fossil use OpenSSL that prevents Fossil from trusting private CA's, or there could be some subtlety in how our private CA administrator issued my Fossil server's certificate.

This took way too much effort to work out, despite having come up in Unix before Slackware even existed, but it's doable:

slackpkg update                     # stock instructions from the Slackware Beginner's Guide
slackpkg install-new
slackpkg upgrade-all
slackpkg clean-system

slackpkg install gcc make           # Fossil build deps
slackpkg install binutils glibc     # not declared as deps of gcc because…? gcc useless else
slackpkg install libmpc             # ditto; may I introduce you to a concept called package dependency declaration?
slackpkg install openssl zlib       # Fossil minimum external libs
slackpkg install linux-headers      # else build against zlib fails; remind me about deps decls…?
slackpkg install tcl                # autosetup's jimsh0 bootstrap step fails; sidestep it
slackpkg install man less groff     # RTFM is difficult without
slackpkg install 'g++'              # only way to get libstdc++, needed by groff; about decls again…?
slackpkg install perl dcron         # needed by update-ca-certificates; fifth verse, same as the first…
slackpkg install ca-certificates    # installs outdated CA roots from late 2023
update-ca-certificates --fresh      # fix that so it groks recent Let's Encrypt certs

cd /tmp
wget --no-check-certificate \       # unavoidable at this stage; we'll fix it
     https://fossil-scm.org/home/uv/fossil-linux-x64-2.25.tar.gz
tar xf fossil-linux-x64-2.25.tar.gz

mkdir ~/museum
cd ~/museum
/tmp/fossil clone \
    https://fossil-scm.org/home \
    fossil-scm.fossil               # needs manual prompting; don't store exception
mkdir -p /usr/local/src/fossil/trunk
cd /usr/local/src/fossil/trunk
/tmp/fossil open \
    ~/museum/fossil-scm.fossil      # prompts about certs again; placate it
./configure && make -j11 && make install

cd /etc/ssl/certs
cat > MyLocalCARoot.crt             # paste text from wherever you have it now
update-ca-certificates --fresh      # incorporate the new one

cd ~/museum
fossil clone https://my-local-fossil-server.example.com/ my-repo.fossil

Huzzah! Despite not having any local TLS setting overrides in Fossil, it doesn't gripe about certificates any more, for several intertwined reasons:

1. We rebuild Fossil from source against Slackware's OpenSSL distribution, baking its custom paths into Fossil.

2. We have updated CA roots installed so we can trust Let's Encrypt certs from https://fossil-scm.org and the like.

3. We added our local CA root to the same store so we can clone from local servers that have certs signed by this local CA root authority.

Thank you for attempting to follow along in my footsteps and reproduce the issue. I'm sorry you had a difficult time with Slackware, but the baseline installation you started with appears to be old and incomplete. The heavily recommended Slackware installation is "full" which gives you all the packages within the selected package series (that word intended to be plural).

The Slackware philosophy has always been for the core packages to not declare dependencies because they're expected to be installed together to create the base system. The goal was for the package system to have as little policy as possible, firmly keeping the admin in the driver's seat. I suppose you could call it the anti-Debian, though in truth its genesis predates Debian's by one month. One popular source for supplemental packages is SlackBuilds.org, which does declare dependencies. There are Slackware derivatives such as Salix which also embrace dependencies, but it's easiest to do the recommended full installation so as to altogether avoid the need for dependencies. This situation is due to Slackware ignoring marketing pressure and focusing on being what its existing user base needs it to be. For example, PAM support was only first added in 15.0, 26 years after it first showed up in Red Hat Linux, and even then primarily because the non-PAM code paths were rotting unmaintained in upstream packages. It'll probably be the same with systemd. See also: no graphical installer; making you run fdisk or cfdisk during installation; requiring you to hand-edit /etc/inittab to enable graphical login; not hiding boot messages; preferring lilo over grub. Slackware spends no effort trying to woo people away from Windows, that's for sure.

update-ca-certificates puts symlinks in /etc/ssl/certs to point to files already on hand, plus concatenates them to form /etc/ssl/certs/ca-certificates.crt, nothing more. Plus, update-ca-certificates already gets run by ca-certificates's package install script, no need to run it as you did, plus your wget failed anyway. Though, I'm super confused about your statement that slackpkg install ca-certificates installs certificates from late 2023. Slackware64-15.0 originally shipped with certificates from 2021 and has most recently been updated to late November 2024. You must be using an old mirror.

Testing in a fresh Slackware64-15.0 installation (complete with 2021 certificates) by running strace --trace=openat -o log wget, the certificate used to validate fossil-scm.org is ISRG_Root_X1.pem. That certificate was generated in 2015, according to openssl x509 -noout -text -in /etc/ssl/certs/ISRG_Root_X1.pem as well as the Let's Encrypt website. Your 2023 certificates are not the problem. Your problem is the official Fossil binary's included OpenSSL being compiled to look in /usr/lib/ssl, which does not exist. /etc/ssl is by far the most common location these days. While /usr/lib/ssl seems to be attributed to old Ubuntu, current Ubuntu uses /etc/ssl as well, and even old Ubuntu had it as a symlink. RHEL 5 uses /etc/pki/tls/certs, whereas RHEL 6 onward also provides /etc/ssl as a symlink to the former. Why is /usr/lib/ssl the default OPENSSLDIR baked into the OpenSSL included with the official Fossil binary, when no current Linux distributions use this location? I wouldn't characterize Slackware's use of /etc/ssl as a "custom" path, given that (1) the Linux FHS doesn't mention certificates and (2) /etc/ssl seems to be the emergent de facto standard.

Using that fresh VM, I repeated your procedure, skipping the package installation bits. I got the Fossil binary from https://fossil-scm.org/ (wget verified without issue), and it also failed to verify the certificate when cloning Fossil's repository. Using strace, I confirmed this is because Fossil looks for /usr/lib/ssl/cert.pem and /usr/lib/ssl/certs which do not exist. There are many methods to get around this:

1. mkdir -p /usr/lib/ssl && ln -s /etc/ssl/certs /usr/lib/ssl
2. export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
3. export SSL_CERT_DIR=/etc/ssl/certs
4. fossil set -global ssl-ca-location /etc/ssl/certs/ca-certificates.crt
5. fossil set -global ssl-ca-location /etc/ssl/certs

And yet, even though all of the above work for me with the official Fossil binary, I still cannot get that binary to trust my in-house CA, despite wget being satisfied with it. This is with a fresh VM with Internet access and running the official binary rather than one I compiled myself. As before, I used strace to confirm it's successfully opening and reading the right certificate file corresponding to my CA, but it's choosing not to trust it.

That's the problem I'm trying to solve.

Addressing your numbered list of intertwined reasons:

1. When it comes to paths, the OpenSSL included with the standard Fossil binary is the odd man out, not Slackware's OpenSSL.
2. Unless your CA roots are older than 2015, you're already set up to trust Let's Encrypt.
3. Please, let's drill into this some more. That's what I care about most.

There is a big difference between how you installed your CA certificate versus what I did. Your putting the file directly into /etc/ssl/certs shouldn't have worked, not with the name you gave it. OpenSSL looks in the certificate directory for a file named XXX.0, where XXX is the subject hash of the server's certificate¹. update-ca-certificates generates these XXX.0 symlinks, but it does so only for certificates in /usr/local/share/ca-certificates and those named within /etc/ca-certificates.conf, which is a list of filenames relative to /usr/share/ca-certificates. It also creates /etc/ssl/certs/ca-certificates.crt, but again that file includes only those that it would have symlinked. Furthermore, unless you set SSL_CERT_FILE or ssl-ca-location to /etc/ssl/certs/ca-certificates.crt, OpenSSL will never read it, bombing as soon as it fails to find XXX.0.

Therefore, putting the certificate into /etc/ssl/certs will only work if you do one of the following:

1. Name the file XXX.0 (certainly not MyLocalCARoot.crt)
2. Manually symlink to the file from XXX.0 (but update-ca-certificates --fresh would delete this symlink)
3. export SSL_CERT_FILE=/etc/ssl/certs/MyLocalCARoot.crt
4. fossil set -global ssl-ca-location /etc/ssl/certs/MyLocalCARoot.crt

Can you use strace -o log --trace=openat on Fossil to see what files it's really opening when it chooses to trust https://my-local-fossil-server.example.com/? Please also double-check also the output of fossil ssl. It would be good to check /etc/slackware-version and the versions of OpenSSL and ca-certificates listed in /var/lib/pkgtools/packages (a.k.a., /var/log/packages). Something isn't adding up, and I'm quite curious why your experience is different from mine.

---

1. ^{^} You can get the subject hash using openssl x509 -noout -subject_hash -in /etc/ssl/certs/MyLocalCARoot.crt

the baseline installation you started with appears to be old and incomplete

I started from this official-looking container image and didn't look to see if anything newer was available. Imagine my shock on discovering that it was shipping GCC 4.8.2 and Clang 3.3!

Why a container? Because I couldn't justify setting up a whole VM just to answer this one question.¹

The above-linked container is based on 14.1, and your reply sparked me to do some searching, which turned up this distinctly less official-looking alternative. Comforted by the fact that it's linked from slackware.com, thus blessed as semi-official, I repeated my steps above, less those that have no ongoing relevance due to improvements in the platform. Alas, nearly every workaround documented above was still required, except that the linux-headers step needed to get zlib builds to work is now apparently kernel-headers.

(I did "nope" out of the recommended install-new step when I saw it installing things like Meson and Qt junk, having zero relevance to this test.)

The only thing I was able to skip was the mystery dependency on libmpc, but I got two new requirements in trade:

# slackpkg install guile gc

I couldn't get past the ./configure step without those.

Having done all this, I am once again able to clone with Fossil over an HTTPS connection encrypted with a custom, locally-minted certificate, signed by a local CA root.

The outdated tech within the openssl-1.1.1zg Slackware ships might be relevant here. Have you tried building against an in-tree copy of OpenSSL 3? (Instructions)

I suppose you could call it the anti-Debian

I was giving Slackware more benefit of the doubt here by comparing it to Alpine, which also handles basics like ensuring that you get binutils and the glibc development files when you install gcc. Characterizing the alternative as "leaving me in the driving seat," seems overly charitable; it's closer to leaving me to fend for myself. I happily left behind the software distribution philosophy of manually chasing dependencies via references in README and INSTALL files in the mid 1990s, when I settled on the Red Hat line.

(Back in those days of buying assortments of cheap install CDs from cdrom.com, as one did, I got one for Slackware, which I installed and used briefly, but as you see, it didn't stick.)

RHEL 6 onward also provides /etc/ssl as a symlink to the former.

True, though the /etc/pki/tls tree remains the default baked into the OpenSSL they distribute, as verified by a fossil ssl command, which means building Fossil binaries on a modern Red Hattish platform won't solve the default path problem.

That sparked me to check my own Fossil container build, based on Alpine:

$ podman run --rm -it --entrypoint fossil docker.io/tangentsoft/fossil ssl
OpenSSL-version:      OpenSSL 3.3.2 3 Sep 2024  (0x030300020)
Trust store location
  SSL_CERT_FILE:      
  SSL_CERT_DIR:       
  ssl-ca-location:    
  OpenSSL-cert-file:  /etc/ssl/cert.pem
  OpenSSL-cert-dir:   /etc/ssl/certs
  Trust store used:   
ssl-identity:      

So yes, one can use that as-is on Slackware without pulling the official binary, provided you're willing to install Podman or Docker first.

I doubt the official binaries will transition over to my build method, though.

update-ca-certificates already gets run by ca-certificates's package install script,

Even after starting over with the Slackware 15.0 container, the wget failed until I either added the --no-check-certificate flag or called update-ca-certificates --fresh. Something is outdated in a fresh install, via this container path at least. The --fresh flag may be the key here.

most recently been updated to late November 2024

Yes, I verified that before resorting to the --fresh step.

sh-5.1# slackpkg info ca-certificates

PACKAGE NAME:  ca-certificates-20241120-noarch-1_slack15.0.txz
…

And yes, this step still requires that you manually install perl and dcron first, within this Slackware 15.0 container.

Unless your CA roots are older than 2015, you're already set up to trust Let's Encrypt.

Let's Encrypt has changed their upstream CA root at least twice since then, most recently in April this year.

Your putting the file directly into /etc/ssl/certs shouldn't have worked, not with the name you gave it.

Granted, my choice of installation location wasn't "nice," admixing my local ones with the default package-provided ones, but quoting the update-ca-certificates man page shipped with the platform, "…all certificates with a .crt extension found below /usr/local/share/ca-certificates are also included as implicitly trusted."

Moving those certs to the recommended location and updating the CA cert store doesn't change anything. It still works as before.

Can you use strace -o log --trace=openat on Fossil…

I believe the relevant output line you're looking for is this one:

openat(AT_FDCWD, "/etc/ssl/certs/1de29165.0", O_RDONLY) = 7

That's a symlink pointing at a generated *.pem file named after the CA root cert I installed under /usr/local/share.

You aren't installing the private key, are you? It needs to be the public certificate half only for this to work.

---

1. ^{^} The chance of me needing to have a Slackware VM sitting around waiting for the next question approaches zero, if you account for distribution lifetimes. If a new OS version will come out by the time the next question comes, why put the time into anything with a long life? Ephemeral is the right answer here, and in today's ecosphere, that means containers.

I have been following this out of curiosity, not because I am any kind of expert but I am reminded of a few lessons learned the hard way some years ago when I set myself the task of setting up a PKI for VPN and SSH keys.

Having done all this, I am once again able to clone with Fossil over an HTTPS connection encrypted with a custom, locally-minted certificate, signed by a local CA root.

I do not think Andy has said that his certificate is like this? Mine was not - the CA/root key was used only to create and sign two sub-CA keys which were then used to create the keys/certificates issued to the "users". The relevant point is that to properly validate the end user certificate you need the CA certificate and the sub-CA certificate i.e. the whole chain, regardless of how long it is. Often the issuer will save you the hassle by bundling the required certificates into one file. You guessed it, I spent hours on that one.

I would not put too much emphasis on the fact that wget works but Fossil does not - I think all this is telling us is that wget is, by default at least, less fussy about checking certificates than Fossil is. Yep, spent hours looking for a non existant bug for similar reasons.

You do not need to put the root certificate in the machine trusted store (unless Fossil insists on this which I guess it will not). In some respects it is better not to, in that if the issuer key is compromised you will be vulnerable only through Fossil, rather than any web site.

One detail which escapes me after all this time is how the file name relates to the certificate(s) it holds. I have an idea it doesn't (i.e. you just read all the files in the directories you are pointed at) but could be wrong about that.

Interesting journey, I hope it gets somewhere.

Trevor

I do not think Andy has said that his certificate is like this?

I don't see how else you read the first few paragraphs of the initial post, but sure, if there's a chain involved, you do need to export the full chain before importing it into another system.

I would not put too much emphasis on the fact that wget works but Fossil does not

They're both linked to the platform-provided libssl.so.1.1 under the test sequence documented above.

Not until someone tries building against an in-tree copy of OpenSSL 3 or LibreSSL would I expect the two to behave differently.

I'd do that myself except that since I have a working configuration, I don't have a falsifiable test to attempt.

EDIT: Keep in mind that the wget and fossil tests in question are targeting different hosts: wget to pull a pre-built Fossil binary from fossil-scm.org to bootstrap the test with, and fossil to clone the repo afterward. I would fully expect a wget from the Fossil server to fail to negotiate a TLS connection in Andy's environment, too.

You do not need to put the root certificate in the machine trusted store

You certainly do, if it is a locally-minted root, with no external validity. Andy acknowledges as much in the initial post, speaking of the fact that other local boxes automatically trust this local CA root, doubtless due to internal IT's automatic rollout of local configurations.

What you do not have to do is copy the CA root's private key outside the certificate minting host, only the matching public certificate.

how the file name relates to the certificate(s) it holds.

There's got to be some type of lookup table, evidenced by the strace test, where it goes straight to the correct certificate; it doesn't iterate over all *.crt or *.pem until it finds a match. My guess is that that hex name I posted above is a prefix of a fingerprint, but another scheme could be in effect here, some type of table lookup, perhaps.

Sorry, it took me several days to reply. This whole time I've been researching and experimenting. I now have solid answers and want to report my findings.

Docker

I wanted to replicate your Docker experience, learning more about it in the process. It's a big boondoggle and major digression, but this information might be useful to someone (even me!) down the road.

Installing sboui

My starting point was a Slackware64-15.0 VM, straight from install media, then my next major goal was installing Docker. To make that easier, I installed sboui to automate downloading and installing SlackBuild packages, respecting dependencies along the way. However, without sboui already installed, I had to build it the "hard" way:

sh -ex <<"EOF"
# Create working space
mkdir -p slackbuild
cd slackbuild

# Download various files
wget https://slackbuilds.org/GPG-KEY
wget https://slackbuilds.org/slackbuilds/15.0/libraries/libconfig.tar.gz{,.asc}
wget https://github.com/hyperrealm/libconfig/archive/v1.7.3/libconfig-1.7.3.tar.gz
wget https://slackbuilds.org/slackbuilds/15.0/system/sboui.tar.gz{,.asc}
wget https://github.com/montagdude/sboui/archive/2.4/sboui-2.4.tar.gz

# Verify SlackBuild scripts
gpg --import --trusted-key 0368EF579C7BA3B6 GPG-KEY
gpg --verify libconfig.tar.gz.asc
gpg --verify sboui.tar.gz.asc

# Build and install libconfig
tar -xf libconfig.tar.gz
cd libconfig
ln -s ../libconfig-1.7.3.tar.gz
MAKEFLAGS=-j`nproc` ./libconfig.SlackBuild
cd ..
mv /tmp/libconfig-1.7.3-x86_64-1_SBo.tgz .
installpkg libconfig-1.7.3-x86_64-1_SBo.tgz

# Build and install sboui
tar xf sboui.tar.gz
cd sboui
ln -s ../sboui-2.4.tar.gz
./sboui.SlackBuild
cd ..
mv /tmp/sboui-2.4-x86_64-1_SBo.tgz .
installpkg sboui-2.4-x86_64-1_SBo.tgz

# Cleanup
cd ..
rm -rf /tmp/SBo slackbuild

# Download SlackBuild repository (must switch protocol from git to https because corporate firewall)
sed -i '/^REPO=.*/s%%REPO=https://git.slackbuilds.org/slackbuilds%' /etc/sboui/sboui-backend.conf
sboui-backend update
EOF

Installing Docker

sboui has a full-screen text user interface (sboui) and a non-interactive command-line tool (sboui-backend). It also provides automatic dependency resolution. Most unfortunately, dependency resolution is handled by the interactive program and not the command-line tool, therefore it can't be scripted. I probably should submit a patch to improve that.

This next procedure has to do something supremely weird. sboui knows Docker indirectly depends on google-go-lang, therefore runs its install first. However, it doesn't run each package install in its own login shell, instead relying on sboui's inherited environment. This is a major problem because google-go-lang creates /etc/profile.d/go.sh which puts the new go binary in your PATH. Yet, that won't be included in sboui's environment which predates the installation of google-go-lang, therefore when sboui tries to build runc (another package needed by Docker), the build fails because it uses GCC go instead. The GCC go version included with Slackware64-15.0 is 1.16.5, whereas google-go-lang provides 1.22.9. There is an incompatible syntax change between these two language versions, thus breaking the build. Curiously, the syntax is within a "comment," yet it's considered fatal, what a world. To work around this absolute mess, set the PATH ahead of time, even though the directory won't exist yet. Making matters even more difficult, the directory name includes the version, which has to be scraped from sboui-backend's output.

1. Run: PATH=$(sboui-backend info google-go-lang | sed -rn '/^google-go-lang-(.*)/s%%/usr/lib64/go\1/go/bin%p'):$PATH sboui
2. Type /docker-cli to search for packages containing "docker-cli" in the name
3. Press Right to move to the right pane
4. Press t to tag the docker-cli package
5. Press i to install
6. Press Enter to confirm
7. Press Enter to confirm dependencies
8. Wait for installation to complete
9. Press Enter to close the information window
10. Press q to quit
11. Run: rm -rf /tmp/sboui.* /tmp/*_SBo.tgz /tmp/SBo

In accordance with Docker's special instructions, I did the following to enable the service:

cat >> /etc/rc.d/rc.local <<"EOF"
if [ -x /etc/rc.d/rc.docker ]; then
    /etc/rc.d/rc.docker start
fi
EOF
cat >> /etc/rc.d/rc.local_shutdown <<"EOF"
if [ -x /etc/rc.d/rc.docker ]; then
    /etc/rc.d/rc.docker stop
fi
EOF
chmod +x /etc/rc.d/rc.docker
groupadd -r -g 281 docker
/etc/rc.d/rc.docker start

I'm just running as root, but if you want to, you can add your user(s) to the docker group with: usermod -a -G docker USERNAME

Running the Container

"docker pull slackware/base" doesn't work because the format is so old. Enabling support for old formats is possible, but why bother?

To run the container, I did:

docker pull vbatts/slackware
docker run -t -i vbatts/slackware

Worked just fine, but it's a very barebones container, no compiler or any of that. I can see why you would want dependency resolution: fleshing out a partial installation. Lacking that feature, I'll just install everything.

Installing Missing Packages

The first step is to get HTTPS working:

slackpkg update
slackpkg install dcron perl ca-certificates

I also had to do some undisclosed magic to negotiate my enterprise's HTTPS implementation. Next, I switched from an HTTP mirror to a nearby HTTPS mirror:

sed -i '/^[^#]/s/^/#/' /etc/slackpkg/mirrors
echo https://mirrors.slackware.com/slackware/slackware64-15.0/ >> /etc/slackpkg/mirrors
slackpkg update
slackpkg install {a,ap,d,l,n,tcl}/\*

Unlike the call to upgradepkg --install-new, the above omits the E (emacs), F (FAQ), K (kernel source), KDE, T (TeX), X, XAP (X applications), XFCE, and Y (games) package series.

The download process could be faster. Each file is a separate invocation of wget, therefore no pipelining is possible. It would be great for slackpkg to gather up all the URLs and pass them all at once to wget, curl, or wget2. At least the installation process uses a multi-threaded xz to decompress. The document I linked above recommends using rsync to clone the Slackware mirror, then mounting it into the container and installing from it. That gets around the slow use of wget, at the cost of cloning more files than I really need. I can fix that by tuning the rsync call, but that takes more work than I'm willing to put in right now.

I don't know why I got a checksum failure when the above installed gc (that's a garbage collector library, not to be confused with GCC), but I was able to fix it with slackpkg reinstall gc.

Slackware

Well, that was fun. I'm supposed to learn more about containers, and this was a great start. Time to move on, though not without a few more comments on Slackware itself. I said it puts you in the driver seat. Now that I see what this particular container looks like, it's like you're also expected to install the engine. I certainly remember cdrom.com, but I also remember my first Slackware install being done from a stack of floppy disks.

Why a container? Because I couldn't justify setting up a whole VM just to answer this one question.

That's fair, but the trouble is neither of the easily-found Slackware containers is ready to run. One is an antique, and the other expects you to complete the installation. For this task, I perhaps should have suggested a live CD right off the bat. That particular live CD is very approachable and does a great job of sprucing up the out-of-the-box experience.

Even after starting over with the Slackware 15.0 container, the wget failed until I either added the --no-check-certificate flag or called update-ca-certificates --fresh. Something is outdated in a fresh install, via this container path at least. The --fresh flag may be the key here.

I suspect that despite the order in which you listed your install commands, you didn't have dcron and perl installed before ca-certificates. Installing ca-certificates runs update-ca-certificates --fresh (see doinst.sh inside the package file itself), but that partially fail without those other two packages. Running it again later without --fresh won't work either because it'll think it's already done and won't try again. update-ca-certificates calls c_rehash, a Perl script, to create the XXX.0 symlinks I talked about. Without those, wget and Fossil won't be able to verify anything¹.

Certificates

Let's Encrypt has changed their upstream CA root at least twice since then, most recently in April this year.

That was a change to their intermediate certificates, not their root, thus we are unaffected.

Granted, my choice of installation location wasn't "nice," admixing my local ones with the default package-provided ones, but quoting the update-ca-certificates man page shipped with the platform, "…all certificates with a .crt extension found below /usr/local/share/ca-certificates are also included as implicitly trusted."

That says /usr/local/share/ca-certificates, but you originally wrote cd /etc/ssl/certs. The former will work, the latter will not. Well, not according to the documentation, it won't. Taking a harder look at c_rehash, now that I know it's there, I see that it could work after all. The line FILE: foreach $fname (grep {/\.(pem)|(crt)|(cer)|(crl)$/} @flist) means it'll trust everything named /etc/ssl/certs/*.{pem,crt,cer,crl}, on top of the documented /usr/local/share/ca-certificates/*.crt files that were symlinked by update-ca-certificates. Okay good, one more mystery solved. You confirm it yourself with openat(AT_FDCWD, "/etc/ssl/certs/1de29165.0", O_RDONLY) = 7, proving that your XXX.0 symlink was created.

The outdated tech within the openssl-1.1.1zg Slackware ships might be relevant here. Have you tried building against an in-tree copy of OpenSSL 3?

I've tried everything with both, and everything works with both. I started with building an in-tree copy, which is nice because I can set --openssldir=/etc/ssl to obtain a binary that'll work everywhere but RHEL 5. Other than that one option, whose effect I can approximate with fossil set -global ssl-ca-location, I get identical behavior with OpenSSL 1.1.1 and OpenSSL 3.

I was able to reproduce your first instance of Fossil not being able to verify the certificate ("needs manual prompting; don't store exception"). Fixing it was a simple matter of: fossil set -global ssl-ca-location /etc/ssl/certs. I then built Fossil from the current trunk cloned from upstream, using nothing but defaults, and the resultant binary got /etc/ssl/certs baked into it just fine. It had no trouble communicating with fossil-scm.org, even after I did fossil unset -global ssl-ca-location.

@MelvaigT's post was extremely helpful and much appreciated. It turns out the certificate I was given was not that of the root, no matter what I was originally told by its administrator. There was a higher-level CA involved. Only when I imported and trusted certificates for both was I able to get Fossil to accept my server.

Getting the certificates was a major hassle in itself, but that's beside the point. I was able to build a fairly sizeable certificate bundle for my organization, which I'll later be deploying to numerous Linux systems so they can all happily run Fossil, wget, and others with HTTPS.

I would fully expect a wget from the Fossil server to fail to negotiate a TLS connection in Andy's environment, too.

To be clear, wget worked for me where Fossil failed. wget cares only about the trusting the CA that signed the site's certificate, whereas Fossil appears to also require the CA's parents to be trusted as well.

What you do not have to do is copy the CA root's private key outside the certificate minting host, only the matching public certificate.

At no point was I given access to private keys for anything other than my Fossil server, so there's no chance of me doing that.

There's got to be some type of lookup table, evidenced by the strace test, where it goes straight to the correct certificate; it doesn't iterate over all *.crt or *.pem until it finds a match. My guess is that that hex name I posted above is a prefix of a fingerprint, but another scheme could be in effect here, some type of table lookup, perhaps.

The subject hash computed from the certificate sent by the server is used as the filename, appended with .0 (or subsequent numbers, in the event of a collision) and located in the certs directory. Alternately, if using a single cert file, the whole file is read.

Now that I have things working, I see from the strace log that Fossil is opening not one but two files in /etc/ssl/certs. Aside from the .0 suffix, the filenames are equal to the subject hashes of the certificates of (1) the CA that signed my Fossil server's certificate and (2) the CA that signed the first CA's certificate.

---

1. ^{^} That's not strictly true, since you can convince them to use the concatenated certificate file instead.

I wanted to replicate your Docker experience

I prefer Podman, and not merely because it was a product of Red Hat before being donated to the CNCF just last month. It's smaller without losing important features, it's designed in closer accord with the Unix philosophy, and it's rootless by default.

The only time I use Docker any more is when building multi-arch images, and then only because Podman's competing scheme for doing that is awkward when you're doing it for platforms that you don't have native builders for.¹ Otherwise, Podman's "farm" feature is a suitable replacement for Docker's multi-arch build capabilities.

I didn't try installing Podman here atop my Slackware container, since it requires dipping into SlackBuilds, which I didn't want to mess with, but you might find its smaller size means fewer dependencies, thus less hassle. If not, then at least you end up with an OCI container engine more in line with Slackware's philosophy.

slackpkg install {a,ap,d,l,n,tcl}/\*

That's clever, but it installs way too much. Worst, to my mind, was when I caught it building a new Linux kernel, an utterly useless artifact inside an OCI container.

I'm supposed to learn more about containers

In that spirit, I offer this diff against Fossil's official Dockerfile to make it build atop Slackware instead of Alpine:

--- Dockerfile
+++ Dockerfile
@@ -1,30 +1,33 @@
 # syntax=docker/dockerfile:1.3
 # See www/containers.md for documentation on how to use this file.
 
 ## ---------------------------------------------------------------------
-## STAGE 1: Build static Fossil binary
+## STAGE 1: Create the base OS layer used for build *and* run stages
+## ---------------------------------------------------------------------
+
+FROM vbatts/slackware:latest AS baseos
+RUN slackpkg update && slackpkg upgrade-all
+RUN slackpkg install dcron perl
+RUN slackpkg install ca-certificates
+
+
+## ---------------------------------------------------------------------
+## STAGE 2: Build static Fossil binary
 ## ---------------------------------------------------------------------
 
-### We don't pin a more stable version of our base layer because we want
-### to build with the latest tools and libraries available in case they
-### fixed something that matters to us since the last build.  Everything
-### below depends on this layer, and so, alas, we toss this container's
-### cache on Alpine's release schedule, roughly once a month.
-FROM alpine:latest AS bld
+FROM baseos AS bld
 WORKDIR /fsl
 
 ### Bake the build-time userland into a base layer so it only changes
 ### when the upstream image is updated or we change the package set.
-RUN set -x                                                             \
-    && apk update                                                      \
-    && apk upgrade --no-cache                                          \
-    && apk add --no-cache                                              \
-         gcc make                                                      \
-         linux-headers musl-dev                                        \
-         openssl-dev openssl-libs-static                               \
-         zlib-dev zlib-static
+### Each line is a coherent set of packages; when not alphabetized,
+### they are ordered deps-first for the item at the tail end; beware
+### second-guessing these sets!
+RUN slackpkg install binutils gcc glibc make tcl
+RUN slackpkg install kernel-headers openssl zlib
+RUN slackpkg install gc guile
 
 ### Build Fossil as a separate layer so we don't have to rebuild the
 ### userland for each iteration of Fossil's dev cycle.
 ###
 ### We must cope with a bizarre ADD misfeature here: it unpacks tarballs
@@ -41,51 +44,29 @@
 ADD $FSLURL $FSLSTB
 RUN set -x                                                             \
     && if [ -d $FSLSTB ] ;                                             \
        then mv $FSLSTB/src . ;                                         \
        else tar -xf src.tar.gz ; fi                                    \
-    && src/configure --static CFLAGS='-Os -s' $FSLCFG && make -j16
-
-
-## ---------------------------------------------------------------------
-## STAGE 2: Pare that back to the bare essentials.
-## ---------------------------------------------------------------------
-
-FROM busybox AS os
-ARG UID=499
-
-### Set up that base OS for our specific use without tying it to
-### anything likely to change often.  So long as the user leaves
-### UID alone, this layer will be durable.
-RUN set -x                                                             \
-    && mkdir e log museum                                              \
-    && echo "root:x:0:0:Admin:/:/false"                   > /e/passwd  \
-    && echo "root:x:0:root"                               > /e/group   \
-    && echo "fossil:x:${UID}:${UID}:User:/museum:/false" >> /e/passwd  \
-    && echo "fossil:x:${UID}:fossil"                     >> /e/group
-
-
-## ---------------------------------------------------------------------
-## STAGE 3: Drop BusyBox, too, now that we're done with its /bin/sh &c
-## ---------------------------------------------------------------------
-
-FROM scratch AS run
-COPY --from=bld --chmod=755           /fsl/fossil /bin/
-COPY --from=os  --chmod=600           /e/*        /etc/
-COPY --from=os  --chmod=1777          /tmp        /tmp/
-COPY --from=os  --chown=fossil:fossil /log        /log/
-COPY --from=os  --chown=fossil:fossil /museum     /museum/
+    && src/configure CFLAGS='-Os -s' $FSLCFG && make -j16
+
+
+## ---------------------------------------------------------------------
+## STAGE 3: Pare that back to the bare essentials.
+## ---------------------------------------------------------------------
+
+FROM baseos AS run
+COPY --from=bld --chmod=755           /fsl/fossil /bin/
 
 
 ## ---------------------------------------------------------------------
 ## RUN!
 ## ---------------------------------------------------------------------
 
+WORKDIR /museum
 ENV PATH "/bin"
 EXPOSE 8080/tcp
-USER fossil
 ENTRYPOINT [ "fossil", "server" ]
 CMD [                       \
     "--create",             \
     "--jsmode", "bundled",  \
     "--user",   "admin",    \
-    "museum/repo.fossil" ]
+    "repo.fossil" ]

In terms of diff line count, it's shorter, but that's an illusion due to the way it handles its layers.

This in turn is why it produces an image 31× larger² than the Alpine version on my x86_64 builder here. I don't know if Slackware is infected with the same issue that prevents other glibc-based Linux distros from building static binaries, but if not, you should be able to pare back that bloat, at a cost of building OpenSSL and zlib from source. Alpine gives you those pre-built for the asking, and because it chases dependencies, the package install steps are shorter.

Build time for this version was 377s here, ⅓ longer than the Alpine build. (Not as bad as I expected, to be honest.)

After applying the patch, the easiest way to try it is with:

mkdir -p ~/museum/test
make CENGINE=podman DCFLAGS="-v $HOME/museum/test:/museum:z" container-run

I suspect that despite the order in which you listed your install commands, you didn't have dcron and perl installed before ca-certificates.

Yes, quite possible. I did not give a simple dump of my shell history above. In my attempt to produce a coherent narrative, I may have reordered some of the commands in a way that matters.

But I must say, if slackpkg chased dependencies for you, this wouldn't matter. What's the point of letting you install packages where the post-install script fails silently due to missing deps? Rather than give me the feeling of having uncommon control over my system, this makes me feel like a human enslaved to the needs of a robot. I'm ecumenical when it comes to tech choices, but once again, same as in the mid-1990s, I'm nope-ing right out of this one.

Further to this point, note that in the Dockerfile patch above, the ca-certificates install step had to be given separately, after the dcron+perl step. If you try to install all three packages in a single command, the certificates still don't get registered properly, even if you list the ca-certificates package last.

---

1. ^{^} For instance, I build iperf3 for ARMv7 to run on one of my routers here. I don't keep development-class ARMv7 hosts sitting around here, preventing me from making effective use of Podman build farms. Ditto 32-bit Intel, which I will continue to build for as long as my public containers keep seeing nonzero download counts during each release cycle.
2. ^{^} 279 MB uncompressed vs 8.9 MB

Podman is what we use here at work, simply because it's what comes with RHEL, but I know not much about it. What I read just now I think is interesting, particularly after seeing just how much went into getting Docker operational. I'm also happy to see it leaving Red Hat's sole control. I'm tempted to come back and try this again with Podman in place of Docker, though I can't justify the time expenditure just now.

Building a new kernel? That certainly didn't happen with me, nor could it. The kernel source package doesn't attempt to build, and the kernel binary package is already built. I suspect you saw it making an initrd. Even so, kernel binaries, kernel modules, and initrds are knees on a fish when it comes to containers.

Thanks for your Dockerfile update. I read through it but will have to revisit another time.

All this put together sums up to Slackware never being designed for containerized deployment, even if it hosts containers alright. While it's nice that it's accidentally usable as a container, more work needs to be done before the user experience can be said to be anything other than awful. Expanding its goals to include containerized deployment would necessarily include making it easier to install a reduced package subset. Installing "everything" is all well and good when it's a bare-metal or VM OS, but containers need to not be gigabytes in size. I don't recall where I saw this, could have been a dream, but I think there was some inkling towards officially adding dependencies to packages. Dependencies have always been a thing with SlackBuild scripts, and there are Slackware derivatives such as Salix which also do dependencies, so there is precedent. For as long as Slackware's goals stop with installation and maintenance of a core set of packages, this is how things will continue to operate. Bringing containers into the mix might help spur the next evolutionary step.

How to add dependencies? Advertising them directly inside the package files would be extremely slow. It seems simplest to extend PACKAGES.TXT, which enumerates a repository. At present it lists package names, locations, sizes, and descriptions. It should be possible for it to also list prerequisites. However, this opens the door to substantial complexity Slackware has avoided for over thirty years. Should the prerequisites simply be names of other packages within the same repository? How about supplemental repositories referring back to packages that ship with the base distribution? Should the names include versions? Should ranges of compatible versions be given? How about conflicts? Identifying when one package supersedes another? Suggested and recommended companion packages? There is a lot of potential to transform into Debian.

Regarding static binaries, I'm able to make them from Slackware. In fact, my main use for Slackware here at work is to be a controlled, scripted environment in which to produce static Fossil and Tcl binaries for Linux and Windows, runnable on all our systems. This whole exchange has shown me that I should eventually dig into improving Slackware's containerization to the point where I can continue to use it for this purpose without having to incur the costs of installation or running in a VM.

What you do not have to do is copy the CA root's private key outside the certificate minting host, only the matching public certificate.

I hope I did not write anything which suggested this might ever be a good idea. To emphasize, private keys need to be kept just that, private. This is the reason for certificate chains rather than just directly signing things - the root CA key is used only on rare occasions under highly controlled conditions. The sort of conditions it would not be practical to apply to, for example, a Fossil server. In short I would say 'must not' rather than 'do not have to'. (This note is really for general consumption, I suspect Warren is well aware of this and was maybe assuming I was not).

You do not need to put the root certificate in the machine trusted store

You certainly do, if it is a locally-minted root, with no external validity.

I am not sure what you mean by 'external validity'. From the point of view of the math / algorithms a self signed certificate produced by you are I is no different to one produced by, say, Verisign. Yes, in the case of Verisign you can go to their web site or many other places to check that the root certificate you have that says it is signed by Verisign is the same, rather than a forgery that looks the same except for the signature (because it is signed with a bogus private key). But that procedure isn't something that helps an SSL library, because it would be a circular argument - how does it know the Verisign.com that it is accessing to check the CA root is the real one rather than one created with the key behind the certificate that you are trying to check?

I would not put too much emphasis on the fact that wget works but Fossil does not

They're both linked to the platform-provided libssl.so.1.1 under the test sequence documented above.

Not until someone tries building against an in-tree copy of OpenSSL 3 or LibreSSL would I expect the two to behave differently.

But using the same library is only a necessary condition, no? Not sufficient. You also need to be calling the same interfaces with the same parameters.

In today's post andy suggests

wget cares only about the trusting the CA that signed the site's certificate, whereas Fossil appears to also require the CA's parents to be trusted as well.

FWIW that sounds like a good call to me. I assume the idea is that the signers certificate would only have been put in the trusted store if somebody had already done the work to chase down and check the root. That does not sound safe to me, so I am with Fossil on this one.

I am reminded of a couple of reasons I like Fossil

1. It is run by a small team who regularly demonstrate they take security seriously.

2. There are folk here like Warren who are prepared to put in a lot of effort to understand an issue just in case it turns out to be Fossil's fault, even though it is not unusual for it to turn out to be user "error" of some kind.

Trevor

No one has suggested, not even by accident or through unclear language, to use private keys in place of certificates. Don't worry about that, it's perfectly clear.

Adding root certificates is required, though most people will be satisfied with the certificates provided by their OS distribution or their IT team. The disagreement comes about when intermediate CAs are involved. Certainly their certificates need to be marked trusted, but what about their parent CAs' certificates?

Concerning the circularity you point out, breaking the cycle is the primary security argument for having a trust store. (There are also technical arguments such as performance and network load.) What's happening is leveraging one form of trust to establish another. You're already implicitly trusting your OS vendor, plus you're trusting whoever has root on your system. Let them provide the list of certificates for CAs you will likewise trust to positively identify web servers and such.

Thanks for the good word about Fossil and its developers. I know it's been years since I myself could be an active contributor, but I am trying to make life changes that would allow me to return to that role.

Thank you also for helping me to get to the bottom of this problem. Thanks as well to @wyoung.

I will create a new reply to my original post summarizing what's left to discuss.

Have you tested what wget does if you now remove the root certificate? Does it still accept the intermediate alone? i.e. is it merely the fact that the intermediate was present by itself to begin with, or was wget really giving it special status because of where it was stored?

Here are my recommendations:

• Update ssl-server.md to explain that "fullchain.pem" must contain not only the server's certificate but also the certificates of all intermediary CAs in the chain of trust.
• Replace the word "certificate" with "certificate chain" in the documentation for http and server/ui.

I don't think this would be right. Doing so may be correct in some situations, but it certainly isn't necessary. The only 'must' is that all certificates in the chain need to be available somehow - the how is up to the user really, unless we are to produce a whole treatise on key management.

I think you may be conflating two different uses of 'trust'.

When you verify a certificate chain what you are doing in essence is checking that the signature on each certificate has been produced using the private key corresponding to the public key given in the parent certificate. The root certificate is only special in the sense that it is its own parent. You signify that you trust the root purely by making it available to the verification process, not by putting it in a particular place.

The 'trust store' is not called that because its contents are to be trusted. It is because we trust the store that its contents cannot be changed without appropriate authority. Obviously you want this protection for root certificates, as one attack on a system is to introduce a bogus root. It does no harm to put other things in there, as it does not give them any special status save (in the case of ssl) making them available to things other than just other instances of Fossil.

Try this thought experiment: what if somebody manages to obtain the private key for the intermediate authority. They can then produce a certificate for a bogus server which will be accepted. What difference will it make where the intermediate certificate comes from, or what status it has?

Trevor

Update ssl-server.md to…

Better?