Fossil

*   **Availability** — nginx is already in most operating system binary
package repositories, so you don’t need to go out of your way to get it.


## Fossil Remote Access Methods

Fossil provides four major ways to access a repository it’s serving
remotely, three of which you can use with nginx:

*   **HTTP** — Fossil has a built-in HTTP server: `fossil server`.
    While this method is efficient and it’s possible to use nginx to
    proxy access to another HTTP server, this option is overkill for our
    purposes.  nginx is itself a fully featured HTTP server, so we will
    choose in this guide not to make nginx reinterpret Fossil’s
    implementation of HTTP.
................................................................................
repository mounted in a different location in the URL scheme.  Here, for
example, we’re saying that the “`example`” repository is hosted under
the `/code` URI on its domains, but that the “`foo`” repo is hosted at
the top level of its domain.  You’ll want to do something like the
former for a Fossil repo that’s just one piece of a larger site, but the
latter for a repo that is basically the whole point of the site.

This script’s automatic restart feature makes Fossil upgrades easy:

       $ cd ~/src/fossil/trunk ; fossil up ; make ; killall fossil ;
         sudo make install ; fslsrv

I’ve written that as a single long command because I keep it in the
history for my Fossil servers, so I can just run it again from history.
You could put it in a shell script instead.



The `killall fossil` step is needed only on OSes that refuse to let you
replace a running binary on disk.

As written, the `fslsrv` script assumes a Linux environment.  It expects
`/bin/bash` to exist, and it depends on non-POSIX tools like `pgrep`.
It shouldn’t be difficult to port to very different systems, like macOS
or the BSDs.


# Configuring Let’s Encrypt, the Easy Way

If your web serving needs are simple, [Certbot][cb] can configure nginx
for you and keep its certificates up to date.  The details are pretty
much as in the Certbot documentation for [nginx on Ubuntu 18.04 LTS
guide][cbnu], except that where they recommend that you use the
first-party Certbot packages, we’ve found that the ones that come with
Ubuntu work just fine.

The primary local configuration you need is to tell nginx how to proxy
certain URLs down to the Fossil instance you started above with the
`fslsrv` script:

      location / {
           include scgi_params;
................................................................................
# Configuring Let’s Encrypt, the Hard Way

If you’re finding that you can’t get certificates to be issued or
renewed using the Easy Way instructions, the problem is usually that
your nginx configuration is too complicated for Certbot’s `--nginx`
plugin to understand. It attempts to rewrite your nginx configuration
files on the fly to achieve the renewal, and if it doesn’t put its
directives in the right locations, the ACME verification steps can fail.







Your author’s configuration, glossed above, is complicated enough that
the current version of Certbot (0.28 at the time of this writing) can’t
cope with it.  That’s the primary motivation for me to write this guide:
I’m addressing the “me” years hence who needs to upgrade to Ubuntu 20.04
or 22.04 LTS and has forgotten all of this stuff. 😉

................................................................................
easier. The parameter file this directive references should be
generated automatically by the Let’s Encrypt package upon installation,
making those parameters unique to your server and thus unguessable. If
the file doesn’t exist on your system, you can create it manually, so:

      $ sudo openssl dhparam -out /etc/letsencrypt/dhparams.pem 2048

Beware, this will take a few minutes of CPU time.



The next section is also optional. It enables [OCSP stapling][ocsp], a
protocol that improves the speed and security of the TLS connection
negotiation.

The next section containing the `ssl_protocols` and `ssl_ciphers` lines
restricts the TLS implementation to only those protocols and ciphers
................................................................................
lock your users out of your site by jumping to HSTS hastily. When you’re
ready, there are [guides you can follow][nest] elsewhere online.


### HTTP-Only Service

While we’d prefer not to offer HTTP service at all, we need to do so for
two reasons, one temporary and the other going forward indefinitely.

First, until we get Let’s Encrypt certificates minted and configured
properly, we can’t use HTTPS yet at all.


Second, the Certbot ACME HTTP-01 challenge used by the Let’s Encrypt
service only runs over HTTP, because it has to work before HTTPS is

working, or after a certificate is accidentally allowed to lapse.  This
is the protocol Let’s Encrypt uses to determine whether we actually have
control over the domains we want our certificate to be minted for.
Let’s Encrypt will not just let you mint certificates for `google.com`
and `paypal.com`!

So, from the second `service { }` block, we include this file to set up
the minimal HTTP service we reqiure, `local/http-certbot-only`:

      listen 80;
      listen [::]:80;
  
................................................................................

We’ll uncomment the `rewrite` and `return` directives below, when we’re
ready to begin testing.


#### Why the Repetition?

You need to do much the same sort of thing as above for each domain name
hosted by your nginx server.


You might being to wonder, then, why I haven’t factored some of those

directives into the included files `local/tls-common` and
`local/http-certbot-only`. For example, why can’t the second HTTP-only
`server { }` block above just be these two lines:


      server_name .foo.net;
      include local/http-certbot-only;

Then in `local/http-certbot-only`, we’d like to say:

      root /var/www/$host;
      access_log /var/log/nginx/$host-http-access.log;
       error_log /var/log/nginx/$host-http-error.log;

Sadly, nginx doesn’t allow variable subtitution into any of these
directives. As I understand it, allowing that would make nginx slower,
so we must largely repeat these directives in each HTTP `server { }`
block.

These configurations are, as shown, as small as I know how to get them.
If you know of a way to reduce some of this repitition, [I solicit your
advice][fd].
................................................................................
         --webroot-path /var/www/foo.net \
             -d foo.net -d www.foo.net

There are two key options here.

First, we’re telling Certbot to use its `--webroot` plugin instead of
the automated `--nginx` plugin. With this plugin, Certbot writes the
ACME HTTP-01 challenge files to the static web document root directory
behind each domain.  For this example, we’ve got two web roots, one of
which holds documents for two different second-level domains
(`example.com` and `example.net`) with `www` at the third level being
optional.  This is a common sort of configuration these days, but you
needn’t feel that you must slavishly imitate it; the other web root is
for an entirely different domain, also with `www` being optional.  Since
all of these domains are served by a single nginx instance, we need to
give all of this in a single command, because we want to mint a single
certificate that authenticates all of these domains.

The second key option is `--dry-run`, which tells Certbot not to do
anything permanent.  We’re just seeing if everything works as expected,
at this point.


### Troubleshooting the Dry Run
................................................................................
If that didn’t work, try creating a manual test:

      $ mkdir -p /var/www/example.com/.well-known/acme-challenge
      $ echo hi > /var/www/example.com/.well-known/acme-challenge/test

Then try to pull that file over HTTP — not HTTPS! — as
`http://example.com/.well-known/acme-challenge/test`. I’ve found that
using Firefox and Safari is better for this sort of thing than Chrome,
because Chrome is more aggressive about automatically forwarding URLs to
HTTPS even if you requested “`http`”.

In extremis, you can do the test manually:

      $ telnet foo.net 80
      GET /.well-known/acme-challenge/test HTTP/1.1
................................................................................
      Last-Modified: Sat, 19 Jan 2019 18:21:54 GMT
      Connection: keep-alive
      ETag: "5c436ac2-4"
      Accept-Ranges: bytes

      hi

You’re looking for that “hi” line at the end and the “200 OK” response





here. If you get a 404 or other error response, you need to look into
your web server logs to find out what’s going wrong.

Note that it’s important to do this test with HTTP/1.1 when debugging a
name-based virtual hosting configuration like this, if the test domain
is one of the secondary names, as in the example above, `foo.net`.




If you’re still running into trouble, the log file written by Certbot
can be helpful.  It tells you where it’s writing it early in each run.



## Step 4: Getting Your First Certificate
................................................................................
document.  If you do not have commit access on the `fossil-scm.org`
repository to update this document as the world changes around it, you
can discuss this document [on the forum][fd].  This document’s author
keeps an eye on the forum and expects to keep this document updated with
ideas that appear in that thread.

[2016]: https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg22907.html

[cb]:   https://certbot.eff.org/
[cbnu]: https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx
[fd]:   https://fossil-scm.org/forum/forumpost/ae6a4ee157
[hsts]: https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
[lja]:  https://en.wikipedia.org/wiki/Logjam_(computer_security)
[mitm]: https://en.wikipedia.org/wiki/Man-in-the-middle_attack
[nest]: https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
[ocsp]: https://en.wikipedia.org/wiki/OCSP_stapling
[qslc]: https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices
[qslt]: https://www.ssllabs.com/ssltest/
[scgi]: https://en.wikipedia.org/wiki/Simple_Common_Gateway_Interface
[vps]:  https://en.wikipedia.org/wiki/Virtual_private_server

*   **Availability** — nginx is already in most operating system binary
package repositories, so you don’t need to go out of your way to get it.


## Fossil Remote Access Methods

Fossil provides four major ways to access a repository it’s serving
remotely, three of which are straightforward to use with nginx:

*   **HTTP** — Fossil has a built-in HTTP server: `fossil server`.
    While this method is efficient and it’s possible to use nginx to
    proxy access to another HTTP server, this option is overkill for our
    purposes.  nginx is itself a fully featured HTTP server, so we will
    choose in this guide not to make nginx reinterpret Fossil’s
    implementation of HTTP.
................................................................................
repository mounted in a different location in the URL scheme.  Here, for
example, we’re saying that the “`example`” repository is hosted under
the `/code` URI on its domains, but that the “`foo`” repo is hosted at
the top level of its domain.  You’ll want to do something like the
former for a Fossil repo that’s just one piece of a larger site, but the
latter for a repo that is basically the whole point of the site.

You might also want another script to automate the update, build, and
deployment steps for new Fossil versions:

       #!/bin/sh
       cd $HOME/src/fossil/trunk
       fossil up
       make -j11
       killall fossil
       sudo make install
       fslsrv

The `killall fossil` step is needed only on OSes that refuse to let you
replace a running binary on disk.

As written, the `fslsrv` script assumes a Linux environment.  It expects
`/bin/bash` to exist, and it depends on non-POSIX tools like `pgrep`.
It should not be difficult to port to systems like macOS or the BSDs.



# Configuring Let’s Encrypt, the Easy Way

If your web serving needs are simple, [Certbot][cb] can configure nginx
for you and keep its certificates up to date. You can follow the Certbot
documentation for [nginx on Ubuntu 18.04 LTS guide][cbnu] as-is, though
we’d recommend one small change: to use the version of Certbot in the
Ubuntu package repository rather than the first-party Certbot package
that the guide recommends.

The primary local configuration you need is to tell nginx how to proxy
certain URLs down to the Fossil instance you started above with the
`fslsrv` script:

      location / {
           include scgi_params;
................................................................................
# Configuring Let’s Encrypt, the Hard Way

If you’re finding that you can’t get certificates to be issued or
renewed using the Easy Way instructions, the problem is usually that
your nginx configuration is too complicated for Certbot’s `--nginx`
plugin to understand. It attempts to rewrite your nginx configuration
files on the fly to achieve the renewal, and if it doesn’t put its
directives in the right locations, the domain verification can fail.

Let’s Encrypt uses the [Automated Certificate Management
Environment][acme] protocol (ACME) to determine whether a given client
actually has control over the domain(s) for which it wants a certificate
minted.  Let’s Encrypt will not blithely let you mint certificates for
`google.com` and `paypal.com` just because you ask for it!

Your author’s configuration, glossed above, is complicated enough that
the current version of Certbot (0.28 at the time of this writing) can’t
cope with it.  That’s the primary motivation for me to write this guide:
I’m addressing the “me” years hence who needs to upgrade to Ubuntu 20.04
or 22.04 LTS and has forgotten all of this stuff. 😉

................................................................................
easier. The parameter file this directive references should be
generated automatically by the Let’s Encrypt package upon installation,
making those parameters unique to your server and thus unguessable. If
the file doesn’t exist on your system, you can create it manually, so:

      $ sudo openssl dhparam -out /etc/letsencrypt/dhparams.pem 2048

Beware, this can take a long time. On a shared Linux host I tried it on
running OpenSSL 1.1.0g, it took about 21 seconds, but on a fast, idle
iMac running LibreSSL 2.6.5, it took 8 minutes and 4 seconds!

The next section is also optional. It enables [OCSP stapling][ocsp], a
protocol that improves the speed and security of the TLS connection
negotiation.

The next section containing the `ssl_protocols` and `ssl_ciphers` lines
restricts the TLS implementation to only those protocols and ciphers
................................................................................
lock your users out of your site by jumping to HSTS hastily. When you’re
ready, there are [guides you can follow][nest] elsewhere online.


### HTTP-Only Service

While we’d prefer not to offer HTTP service at all, we need to do so for
two reasons:

*   The temporary reason is that until we get Let’s Encrypt certificates
    minted and configured properly, we can’t use HTTPS yet at all.

*   The ongoing reason is that the Certbot [ACME][acme] HTTP-01
    challenge used by the Let’s Encrypt service only runs over HTTP. This is
    not only because it has to work before HTTPS is first configured,
    but also because it might need to work after a certificate is
    accidentally allowed to lapse, to get that server back into a state


    where it can speak HTTPS safely again.


So, from the second `service { }` block, we include this file to set up
the minimal HTTP service we reqiure, `local/http-certbot-only`:

      listen 80;
      listen [::]:80;
  
................................................................................

We’ll uncomment the `rewrite` and `return` directives below, when we’re
ready to begin testing.


#### Why the Repetition?

These `server { }` blocks contain several directives that have to be
either completely repeated or copied with only trivial changes when
you’re hosting multiple domains from a single server.


You might then wonder, why haven’t I factored some of those directives
into the included files `local/tls-common` and

`local/http-certbot-only`? Why can’t the HTTP-only `server { }` block
above be just two lines? That is, why can I not say:

      server_name .foo.net;
      include local/http-certbot-only;

Then in `local/http-certbot-only` say:

      root /var/www/$host;
      access_log /var/log/nginx/$host-http-access.log;
       error_log /var/log/nginx/$host-http-error.log;

Sadly, nginx doesn’t allow variable subtitution into these particular
directives. As I understand it, allowing that would make nginx slower,
so we must largely repeat these directives in each HTTP `server { }`
block.

These configurations are, as shown, as small as I know how to get them.
If you know of a way to reduce some of this repitition, [I solicit your
advice][fd].
................................................................................
         --webroot-path /var/www/foo.net \
             -d foo.net -d www.foo.net

There are two key options here.

First, we’re telling Certbot to use its `--webroot` plugin instead of
the automated `--nginx` plugin. With this plugin, Certbot writes the
[ACME][acme] HTTP-01 challenge files to the static web document root
directory behind each domain.  For this example, we’ve got two web
roots, one of which holds documents for two different second-level
domains (`example.com` and `example.net`) with `www` at the third level
being optional.  This is a common sort of configuration these days, but
you needn’t feel that you must slavishly imitate it; the other web root
is for an entirely different domain, also with `www` being optional.
Since all of these domains are served by a single nginx instance, we
need to give all of this in a single command, because we want to mint a
single certificate that authenticates all of these domains.

The second key option is `--dry-run`, which tells Certbot not to do
anything permanent.  We’re just seeing if everything works as expected,
at this point.


### Troubleshooting the Dry Run
................................................................................
If that didn’t work, try creating a manual test:

      $ mkdir -p /var/www/example.com/.well-known/acme-challenge
      $ echo hi > /var/www/example.com/.well-known/acme-challenge/test

Then try to pull that file over HTTP — not HTTPS! — as
`http://example.com/.well-known/acme-challenge/test`. I’ve found that
using Firefox or Safari is better for this sort of thing than Chrome,
because Chrome is more aggressive about automatically forwarding URLs to
HTTPS even if you requested “`http`”.

In extremis, you can do the test manually:

      $ telnet foo.net 80
      GET /.well-known/acme-challenge/test HTTP/1.1
................................................................................
      Last-Modified: Sat, 19 Jan 2019 18:21:54 GMT
      Connection: keep-alive
      ETag: "5c436ac2-4"
      Accept-Ranges: bytes

      hi

You type the first two lines at the remote system, plus the doubled
“Enter” to create the blank line, and you get something back that
hopefully looks like the rest of the text above.

The key bits you’re looking for here are the “hi” line at the end — the
document content you created above — and the “200 OK” response code. If
you get a 404 or other error response, you need to look into your web
server logs to find out what’s going wrong.

Note that it’s important to do this test with HTTP/1.1 when debugging a
name-based virtual hosting configuration like this. Unless you test only

with the primary domain name alias for the server, this test will fail.
Using the example configuration above, you can only use the
easier-to-type HTTP/1.0 protocol to test the `foo.net` alias.

If you’re still running into trouble, the log file written by Certbot
can be helpful.  It tells you where it’s writing it early in each run.



## Step 4: Getting Your First Certificate
................................................................................
document.  If you do not have commit access on the `fossil-scm.org`
repository to update this document as the world changes around it, you
can discuss this document [on the forum][fd].  This document’s author
keeps an eye on the forum and expects to keep this document updated with
ideas that appear in that thread.

[2016]: https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg22907.html
[acme]: https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment
[cb]:   https://certbot.eff.org/
[cbnu]: https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx
[fd]:   https://fossil-scm.org/forum/forumpost/ae6a4ee157
[hsts]: https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
[lja]:  https://en.wikipedia.org/wiki/Logjam_(computer_security)
[mitm]: https://en.wikipedia.org/wiki/Man-in-the-middle_attack
[nest]: https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
[ocsp]: https://en.wikipedia.org/wiki/OCSP_stapling
[qslc]: https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices
[qslt]: https://www.ssllabs.com/ssltest/
[scgi]: https://en.wikipedia.org/wiki/Simple_Common_Gateway_Interface
[vps]:  https://en.wikipedia.org/wiki/Virtual_private_server