Fossil

Serving via nginx on Debian and Ubuntu
Login

Serving via nginx on Debian and Ubuntu

This document is an extension of the platform-independent SCGI instructions, which may suffice for your purposes if your needs are simple.

Here, we add more detailed information on nginx itself, plus details about running it on Debian type OSes. This document was originally written for and tested on Debian 10 (Buster) and Ubuntu 20.04, which were common Tier 1 OS offerings for virtual private servers at the time. The same configuration appears to run on Ubuntu 22.04 LTS without change. This material may not work for older OSes. It is known in particular to not work as given for Debian 9 and older!

We also cover adding TLS to the basic configuration, because several details depend on the host OS and web stack details. Besides, TLS is widely considered part of the baseline configuration these days.

Benefits

This scheme is considerably more complicated than the standalone HTTP server and CGI options. Even with the benefit of this guide and pre-built binary packages, it requires quite a bit of work to set it up. Why should you put up with this complexity? Because it gives many benefits that are difficult or impossible to get with the less complicated options:

Fossil Service Modes

Fossil provides four major ways to access a repository it’s serving remotely, three of which are straightforward to use with nginx:

SCGI it is, then.

Installing the Dependencies

The first step is to install some non-default packages we’ll need. SSH into your server, then say:

$ sudo apt install fossil nginx

You can leave “fossil” out of that if you’re building Fossil from source to get a more up-to-date version than is shipped with the host OS.

Running Fossil in SCGI Mode

For the following nginx configuration to work, it needs to contact a background Fossil instance speaking the SCGI protocol. There are many ways to set that up, such as with systemd on mainstream Linux distros. Another way is to containerize your repository servers, then use the fslsrv wrapper for Podman to generate systemd units for use by the front-end proxy.

However you do it, you need to match up the TCP port numbers between it and those in the nginx configuration below.

Configuration

On Debian and Ubuntu systems the primary user-level configuration file for nginx is /etc/nginx/sites-enabled/default. I recommend that this file contain only a list of include statements, one for each site that server hosts:

include local/example.com
include local/foo.net

Those files then each define one domain’s configuration. Here, /etc/nginx/local/example.com contains the configuration for *.example.com and its alias *.example.net; and local/foo.net contains the configuration for *.foo.net.

The configuration for our example.com web site, stored in /etc/nginx/sites-enabled/local/example.com is:


server {
    server_name .example.com .example.net "";
    include local/generic;
    include local/code;

    access_log /var/log/nginx/example.com-https-access.log;
    error_log /var/log/nginx/example.com-https-error.log;

    # Bypass Fossil for the static documentation generated from
    # our source code by Doxygen, so it merges into the embedded
    # doc URL hierarchy at Fossil’s $ROOT/doc without requiring that
    # these generated files actually be stored in the repo.  This
    # also lets us set aggressive caching on these docs, since
    # they rarely change.
    location /code/doc/html {
        root /var/www/example.com/code/doc/html;

        location ~* \.(html|ico|css|js|gif|jpg|png)$ {
            add_header Vary Accept-Encoding;
            access_log off;
            expires 7d;
        }
    }

    # Redirect everything under /code to the Fossil instance
    location /code {
        include local/code;

        # Extended caching for URLs that include unique IDs
        location ~ "/(artifact|doc|file|raw)/[0-9a-f]{40,64}" {
            add_header Cache-Control "public, max-age=31536000, immutable";
            include local/code;
            access_log off;
        }

        # Lesser caching for URLs likely to be quasi-static
        location ~* \.(css|gif|ico|js|jpg|png)$ {
            add_header Vary Accept-Encoding;
            include local/code;
            access_log off;
            expires 7d;
        }
    }
}

As you can see, this is a pure extension of the basic nginx service configuration for SCGI, showing off a few ideas you might want to try on your own site, such as static asset proxying.

You also need a local/code file containing:

include scgi_params;
scgi_pass 127.0.0.1:12345;
scgi_param SCRIPT_NAME "/code";

We separate that out because nginx refuses to inherit certain settings between nested location blocks, so rather than repeat them, we extract them to this separate file and include it from both locations where it’s needed. You see this above where we set far-future expiration dates on files served by Fossil via URLs that contain hashes that change when the content changes. It tells your browser that the content of these URLs can never change without the URL itself changing, which makes your Fossil-based site considerably faster.

Similarly, the local/generic file referenced above helps us reduce unnecessary repetition among the multiple sites this configuration hosts:

root /var/www/$host;

listen 80;
listen [::]:80;

charset utf-8;

There are some configuration directives that nginx refuses to substitute variables into, citing performance considerations, so there is a limit to how much repetition you can squeeze out this way. One such example are the access_log and error_log directives, which follow an obvious pattern from one host to the next. Sadly, you must tolerate some repetition across server { } blocks when setting up multiple domains on a single server.

The configuration for foo.net is similar.

See the nginx docs for more ideas.

Proxying HTTP Anyway

Above, we argued that proxying SCGI is a better option than making nginx reinterpret Fossil’s own implementation of HTTP. If you want Fossil to speak HTTP, just set Fossil up as a standalone server. And if you want nginx to provide TLS encryption for Fossil, proxying HTTP instead of SCGI provides no benefit.

However, it is still worth showing the proper method of proxying Fossil’s HTTP server through nginx if only to make reading nginx documentation on other sites easier:

location /code {
    rewrite ^/code(/.*) $1 break;
    proxy_pass http://127.0.0.1:12345;
}

The most common thing people get wrong when hand-rolling a configuration like this is to get the slashes wrong. Fossil is sensitive to this. For instance, Fossil will not collapse double slashes down to a single slash, as some other HTTP servers will.

Allowing Large Unversioned Files

By default, nginx only accepts HTTP messages up to a meg in size. Fossil chunks its sync protocol such that this is not normally a problem, but when sending unversioned content, it uses a single message for the entire file. Therefore, if you will be storing files larger than this limit as unversioned content, you need to raise the limit. Within the location block:

# Allow large unversioned file uploads, such as PDFs
client_max_body_size 20M;

Integrating fail2ban

One of the nice things that falls out of proxying Fossil behind nginx is that it makes it easier to configure fail2ban to recognize attacks on Fossil and automatically block them. Fossil logs the sorts of errors we want to detect, but it does so in places like the repository’s admin log, a SQL table, which fail2ban doesn’t know how to query. By putting Fossil behind an nginx proxy, we convert these failures to log file form, which fail2ban is designed to handle.

First, install fail2ban, if you haven’t already:

sudo apt install fail2ban

We’d like fail2ban to react to Fossil /login failures. The stock configuration of fail2ban only detects a few common sorts of SSH attacks by default, and its included (but disabled) nginx attack detectors don’t include one that knows how to detect an attack on Fossil. We have to teach it by putting the following into /etc/fail2ban/filter.d/nginx-fossil-login.conf:

[Definition]
failregex = ^<HOST> - .*POST .*/login HTTP/..." 401

That teaches fail2ban how to recognize the errors logged by Fossil.

Then in /etc/fail2ban/jail.local, add this section:

[nginx-fossil-login]
enabled = true
logpath = /var/log/nginx/*-https-access.log

The last line is the key: it tells fail2ban where we’ve put all of our per-repo access logs in the nginx config above.

There’s a lot more you can do, but that gets us out of scope of this guide.

Adding TLS (HTTPS) Support

One of the many ways to provide TLS-encrypted HTTP access (a.k.a. HTTPS) to Fossil is to run it behind a web proxy that supports TLS. Because one such option is nginx, it’s best to delegate TLS to it if you were already using nginx for some other reason, such as static content serving, with only part of the site being served by Fossil.

The simplest way by far to do this is to use Let’s Encrypt’s Certbot, which can configure nginx for you and keep its certificates up to date. You need but follow their nginx on Ubuntu 20 guide. We had trouble with this in the past, but either Certbot has gotten smarter or our nginx configurations have gotten simpler, so we have removed the manual instructions we used to have here.

You may wish to include something like this from each server { } block in your configuration to enable TLS in a common, secure way:

# Tell nginx to accept TLS-encrypted HTTPS on the standard TCP port.
listen 443 ssl;
listen [::]:443 ssl;

# Reference the TLS cert files produced by Certbot.
ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

# Load the Let's Encrypt Diffie-Hellman parameters generated for
# this server.  Without this, the server is vulnerable to Logjam.
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

# Tighten things down further, per Qualys’ and Certbot’s advice.
ssl_session_cache shared:le_nginx_SSL:1m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_session_timeout 1440m;

# Offer OCSP certificate stapling.
ssl_stapling on;
ssl_stapling_verify on;

# Enable HSTS.
include local/enable-hsts;

The HSTS step is optional and should be applied only after due consideration, since it has the potential to lock users out of your site if you later change your mind on the TLS configuration. The local/enable-hsts file it references is simply:

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

It’s a separate file because nginx requires that headers like this be applied separately for each location { } block. We’ve therefore factored this out so you can include it everywhere you need it.

The OCSP step is optional, but recommended.

You may find Qualys’ SSL Server Test helpful in verifying that you have set all this up correctly, and that the configuration is strong. We’ve found their best practices doc to be helpful. As of this writing, the above configuration yields an A+ rating when run on Ubuntu 22.04.01 LTS.

Return to the top-level Fossil server article.