Could we please add an option to fossil server to bind to a specific ip instead of either "all" or "127.0.0.1" (implicitely for ui)

There could be hosts with plenty of ip adresses, also "lo" often allows for more than just 127.0.0.1.

It would be nice to restrict that "bind ip", if wanted.

Additionally: could we probably allow binding to a unix domain socket instead of a network as well?

Sometimes, for use cases where the questions you are asking are important, it turns out that running fossil behind a reverse proxy is a better solution than relying on more complex network architecture being provided by a SCM system.

Just as a thought.

thanks for the quick response.

The Unix Domain socket indeed falls only into the categories "nice to have" and "technically interesting". On a multihomed system with hundreds of users I'd feel better with that. Even only listening on 127.0.0.1 would make it typically accessible for all local users, adding the full burden of setting up all kinds of security, users and such only for small all-day-personal repos. But if it's too much work: kick it. It's esoteric.

The -B <BIND_IP> alongside -P <PORT> for fossil server instead is probably only a one (or some?) line patch, and considered useful. My typical servers and routers have more than one interface to different VLANs. Yes, firewall and reverse proxy would also work. But adding a -B <BIND_IP> is most probably really peanuts.

Please note that cmd_webserver() already takes zIpAddr as parameter. Oh ... wait ... let me read the code closely! What's THAT?

It seems -P already supports this -- it is only not documented! fossil server -P IP:PORT is working.

OK, I'm changing my post. My wish is already fulfilled, if you like, please document it!

Thanks and have a nice weekend!

It's already there.

Ah, you're right!

fossil server --port 192.168.0.35:8080 ~/fossil-repos/warren.fossil &

Will only listen on 8080 for that IP address.

Unix domain sockets are a necessary part for me and I very much wish they would be available.

Please add support for unix domain sockets. It is not just a technically interesting thing but a real need.

What would be the use case for it? How are you even accessing a UNIX socket from your browser? As if all you want to do is sync with a repo, you can just do that via SSH, so the only reason you'd need the server is if you want to use the web inteface. And I don't see how that works with a browser over a UNIX socket.

Those who care about security like the fact unix domain sockets¹ can keep miscreants out. The communication happens in the OS kernel thus tools like WireShark can’t scarf up your data.

Back when I was setting up a system I encountered this issue. Have app A talk to app B over an IP:port or use this thing called a unix domain socket? Not a lot of reading on the Internet was required to convince me.

• More secure. In the kernel.
• File permissions are honored, thus being able to prevent other (not authorized) from accessing it.
• Possibly faster. Some claimed it is. Another claim was it uses less RAM.
• Less having to track port numbers in use, reducing complexity.

Anyone who wants to host fossil repos (more securely) should be in favor unix domain sockets.

And I don't see how that works with a browser over a UNIX socket.

Correct. Browsers, at least the one’s I’ve used, do not communicate with the host over a unix domain socket.

In my exploring I’ve found quite a few open source apps offer unix domain sockets. A bunch of them have those permissive licenses which are very good. Thus their code can hopefully serve as a template if not be directly incorporated.

This is not meant to denigrate the great work already poured into fossil.

But adding unix domain sockets so it can communicate with other software more securely² would be another feather in its cap. ;-)

1. ^{^} The Wikipedia entry for Unix domain socket claims they’re part of the POSIX standard. Not esoteric.
2. ^{^} Not to mention it might be faster and may consume less RAM.

With the understanding that I'm not against this idea in principle, I nevertheless find your arguments in favor lacking:

tools like WireShark can’t scarf up your data.

Only for the insufficiently motivated.

In the kernel.

So is a localhost TCP socket.

File permissions are honored

Yes, that is in principle useful. Useful in practice for Fossil in particular, separate question.

able to prevent other (not authorized) from accessing it.

Who is "other," and why does he/it/they have a login on your Fossil server?

No login, no meaningful distinction between a localhost socket and a Unix domain socket, from a security standpoint.

Possibly faster.

Maybe back in the SVR4 days. I expect that Linux plucked all that low-hanging fruit decades ago. The kernel is in a fine position to realize that two equivalent constructs produced by disparate APIs can be handled the same way internally.

Atop that, realize that you're putting all this at the end of an Internet bottleneck in a lot of cases, where tiny differences are swamped.¹

it uses less RAM.

Sure, on the order of ~4-8k for per-connection socket buffers.

It's only going to matter if you're still trying to solve the C10k problem on a single box, like it's 1999 again.

permissive licenses…serve as a template

Lack of knowledge of how to initialize a Unix domain socket is not a blocker. Anyone with a copy of the Stevens books will have this, and the rest can web-search it from open sources. This is not a deep secret.

1. ^{^} If it were not so, all these highly-successful interpreted language web stacks from Perl to Node would never have gotten off the ground.

I expect that Linux plucked all that low-hanging fruit decades ago. The kernel is in a fine position to realize that two equivalent constructs produced by disparate APIs can be handled the same way internally.

Unfortunately, it does not.

One can easily see how TCP over localhost performs compared to UD sockets by taking their local X11-enabled system, removing -nolisten tcp from the X server startup script, restarting X and then trying to use any X-enabled app by running something like

$ DISPLAY=localhost:6000 that-app

to force the app to connect to the server via TCP.

The results are really abysmal for typical Gtk- and Qt-enabled apps which use font smoothing (which requires transmission of bitmaps underlying individual glyphs over the wire) :-)

Honestly, I'm not sure a kernel can sensibly "transparently" replace localhost TCP with UD sockets but I'm not an expert either.

OTOH, I do not expect that much traffic to Fossil and back to make the distinction really noticeable.

That test tells you more about X than it does about sockets.

Terminal 1:

$ rm ~/tmp/test.sock ; nc -lU ~/tmp/test.sock

Terminal 2:

$ time dd if=/dev/zero bs=8k count=64k | nc -U ~/tmp/test.sock
65536+0 records in
65536+0 records out
536870912 bytes transferred in 9.878083 secs (54349707 bytes/sec)
dd if=/dev/zero bs=8k count=64k  0.08s user 0.93s system 10% cpu 9.881 total
nc -U ~/tmp/test.sock  0.64s user 2.10s system 27% cpu 9.882 total

Terminal 1:

$ nc -l 8000

Terminal 2:

$ time dd if=/dev/zero bs=8k count=64k | nc localhost 8000   
65536+0 records in
65536+0 records out
536870912 bytes transferred in 8.860171 secs (60593742 bytes/sec)
dd if=/dev/zero bs=8k count=64k  0.07s user 0.71s system 8% cpu 8.863 total
nc localhost 8000  0.44s user 1.17s system 18% cpu 8.889 total

TCP ran a bit faster and with a smidge less CPU than a Unix domain socket in this test.

I did these tests above on a macOS host, then repeated it in a Linux VM, where I got similar results.

Very interesting, thanks for the heads-up!

Dear Warren, I’m sorry if I kicked up a lot of dust and sounded offensive. You are far more knowledgeable about programming and computers than I am.

Sorry about the absence but later in the day I replied, I accidentally ate something which really disagreed with me and I think I had a small case of food poisoning. I was out of commission for a while. Now I am finally trying to catch up.

With regards to attackers, I agree there are extreme cases, but I consider there to be some amount of work that’s worth it and then at some point, I agree, there are diminishing returns.

Who is "other," and why does he/it/they have a login on your Fossil server?

$ man chmod. User-owner, group, other. Within unix file systems each file and folder (aka directory) has ownership and permissions associated them. Thus the now default permissions for the socket: 660. Let the owner and group both read and write to the socket, but keep others out. Some people use “world” instead of “other”.

It’s a matter of reducing attack surface.

Browsers ... do not communicate with the host over a unix domain socket.

Fossil uses sockets for two purposes:

1. Listen for and respond to HTTP requests from web browsers

2. Listen for HTTP sync requests from other Fossil instances.

You've already stated that (1) won't work with unix domain sockets because web browsers don't support it. We could support unix domain sockets for (2), but if both the client and the server are on the same host, it seems like it would be simpler to use the "file:" scheme on the sync URL and avoid sockets all together. What would be the point of having the extra complication of unix domain sockets?

So I still don't see a use-case for unix domain sockets in Fossil.

The use case is running a fossil server behind a reverse proxy and wanting to take advantage of already discussed security as well as performance features.

I'd use it the moment it's available. Can live without it though.

That is a reasonable use-case that I had not thought of. So now on trunk there is a new option "--unix-socket NAME" option for the "fossil server" command, that causes it to listen on unix-domain socket NAME instead of on some TCP/IP port.

Since this is your idea, Georg, I call upon you to compile the latest trunk check-in, test it out on your reverse proxy, and let us all know whether or not it works.

Thanks for the honor.

You can find the working result at: https://at.magma-soft.at/fossil/

There are two inconveniences for deployment though:

1. Fossil creates the socket when it does not exist, but it does not delete it when exiting.
2. The socket is created with permissions 755, that's probably the default.

The first is a minor issue. It'd help "early fail", since in case of fossil server shutdown the proxy would not find the socket anymore which is a clearer indication then hanging on a vanished socket.

The second issue is more important. One should be able to specify the mode of the socket for taking advantage of privilege separation.

Other server software solves this by allowing to specify the file mode of the socket with e.g. --socket-mode MODE, where MODE is specified e.g. as an octal number representing the required permissions.

As an example illustrating the background I'll explain my setup:

My proxy process runs as user nsd but with group www-data.

My fossil server runs as user fossil with --unix-socket /run/fossil/socket. The directory /run/fossil is owned by user fossil and group www-data, with the group sticky bit set, mode: 02750.

When fossil creates the socket it is owned by fossil but the group is www-data because of the group sticky bit. However the proxy can only read from the socket, because of the r-x permissions for "group" and "other".

I have to set the permissions of /run/fossil/socket manually to 0660 after fossil is started.

Then the fossil process can read from and write to the socket by virtue of the owner permissions and the proxy process can do the same by virtue of the group permissions. No other user or group can interact with the fossil process.

With fossil server --unix-socket /run/fossil/socket --socket-mode 0660 ... the correct permissions would be set directly by fossil on startup.

Other software also allows to specify the owner and group of the socket, but this requires root privileges and as a consequence being able to drop privileges after setting up the socket. These complications can be avoided by the presented setup.

Other software also allows to specify the owner and group of the socket, but this requires root privileges and as a consequence being able to drop privileges after setting up the socket. These complications can be avoided by the presented setup.

Fossil already detects if it is running as root and drops root privilege (after putting itself into a chroot jail, unless --nojail is specified) prior to reading any content off of the socket. So setting the owner and group of the socket is a reasonable thing to do. What are the usual command-line option or options to achieve that?

I'm tempted to make the argument of --unix-socket a JSON (or JSON5) object that contains all of the information you need. Like this:

fossil server --unix-socket '{name:"/run/fossil/socket",mode:"660",owner:"nsd",group:"www-data"}'

Is that completely unreasonable and un-unix-like? What would Dennis Richie think? If the argument to --unix-socket starts with "{" then it is interpreted as a JSON object; otherwise it is just a filename.

Another idea is to allow the argument to --unix-socket be a file: URI with query parameters:

fossil server --unix-socket file:///run/fossil/socket?mode=0666&owner=nds&group=www-data

Note that the URI is shorter than the JSON5 objects.

Just use --socket-owner OWNER with the following common pattern for OWNER:

• If it is of the form user:group set the owner to user and the group to group.
• If it is of the form user set the owner to user and the group to the primary group of user.

Ignore (or error out) if not running as root. In this case setting to anything else then the current process' user will fail anyway.

If you use mode 0660 as default --socket-mode and --socket-owner can be omitted in common scenarios.

I'd prefer not to have them munged together neither into JSON nor URI query parameters. It requires quoting in the shell and makes the command line overly verbose.

Other software:

PHP-FPM uses:

• listen.mode MODE
• listen.owner USER
• listen.group GROUP

UWSGI uses:

• chmod-socket MODE
• chown-socket USER:GROUP

On a branch check-in are changes as follows:

• The "--unix-socket NAME" option is renamed to "--socket-name NAME" for consistency with the new options in the next bullet point.

• Added "--socket-mode MODE" and "--socket-owner USER".

• Try to unlink the socket on close.

Everything seems to be working except for --socket-owner. Somebody please have a look and tell me what I'm doing wrong. There is a debugging printf() in the code showing that the parameters to fchown() are correct, and the fchown() should call fossil_fatal() if something goes wrong. And yet, the ownership of the socket is not being changed.

You must use chown().

1 led me to 2 which explains why.

Thanks for chasing that down.

But doesn't this undermine the claims of better security? There's a clear race condition here.

I suppose it's partly why Unix sockets persist after both apps connected to them die: to hold the permissions in the filesystem so there is only a first-use problem, not a perennial race.

As far as I can tell, the only thing this improves on over TCP localhost sockets is an infinitely wider namespace, compared to the 1024-49151 range non-privileged listening sockets are supposed to use.

As far as I can tell, the only thing this improves on over TCP localhost sockets is an infinitely wider namespace, compared to the 1024-49151 range non-privileged listening sockets are supposed to use.

Well, unix domain sockets have access restrictions even on localhost - IP sockets do not. There could be a race-condition on create, but it is avoidable (by precreating the socket or creating it in controlled directory).

Fossil does fchmod() first, so the permissions are already locked down. When the filesystem inode is created (by the bind() system call), the owner and group are both "root". And since the default permissions are 0660, that means only root can access the socket before the ownership change. So I don't see a way for an attacker to access the socket during the race.

Did I overlook something?

FWIW: The changes are now on trunk. Georg → please try it out and post your evaluations.

Did I overlook something?

No, it seems fine in that order. Delicate in the face of future maintenance, but fine today.

I'll keep doing TCP localhost sockets in production, both because of the benchmark results above and that they seem to play better with container runtimes. I do expect a bind-mounted Unix domain socket works across that boundary, but it seems like excessively clever magic to rely on it.

Container images building as I write this, pushed shortly.

Owner and group are now set correctly. Unfortunately neither the default permissions, nor permissions given with --socket-mode are applied.

I guess the same rationale as for fchown() applies also to fchmod():

You'll have to use chmod() for --socket-mode.

What OS are you using? When I do:

./fossil server --socket-name ./sock1 --socket-mode 0600 &
ls -l ./sock1

I see as output:

srw-------  1 drh drh        0 Aug  6 22:51 sock1

This is on Ubuntu. What are you doing that the mode is not being applied?

I had umask set to 0022. After correcting umask the socket mode is set correctly. This is on Debian Linux.

Sorry for the noise.

As far as I can tell, the only thing this improves on over TCP localhost sockets is an infinitely wider namespace, compared to the 1024-49151 range non-privileged listening sockets are supposed to use.

Yes!

This is another very important point; even including up to 65,535.

Keeping track of, or testing, ports in use is a pain compared to just using existing names provided by a user or a random string to create a socket in a folder.

Deleting the socket on exit is tricky because:

1. The server might have dropped root privilege in between the time when it created the socket and when it exits.

2. The server might have entered a chroot jail after creating the socket and the socket might be somewhere outside of that jail.

Just try and ignore failure.

(1) If the fossil process has owner access to the socket it is cleaned up, if not, (2) the one with owner access has to take care. Looks sensible to me.

1. Even if privileges are dropped, the fossil process will typically be set as the --socket-owner and succeed with deletion. Otherwise: how would fossil access the socket anyway?
2. This only seems to work with abstract sockets (never heard of them until now, see [1]) Do you use them in this case?

When chroot'ing you can try to remove the chroot directory prefix from the socket path before deletion. Stripping that much characters should do. If the socket is inside the chroot it might get deleted, otherwise rule (2) enters.

[1]: https://unix.stackexchange.com/questions/317319/x-org-working-with-no-socket-in-chroot

Richard,

Thank you for your work on this!

But I’m having a problem.

It seems fossil deletes the socket after it’s used once.

Checked:

• fossil and the other binary are allowed in the directory holding the socket.
• fossil and the other app have read/write permissions to the socket. i.e. it’s 660.
• I load the site in my browser; it makes a successful initial connection. fossil sends me the directory of repos and I can see them, like I’m used to. But...
• The socket file is now gone!
• Click a repo or reload the page and the browser gets an error message.
• Quit fossil and open it up again.
• Click a repo link.
• fossil serves up the one request and stops. It appears only the first HTML comes through, no CSS or JS.
• The socket file is now gone!

Fossil as of [faf4b4718d]¹ [668fefe796] and sqlite-snapshot-202408161851.tar.gz on Darwin.

I’ve tried launching fossil a number of ways and it always deletes the socket after it’s used once.

No --errorlog file is created. I also tried not putting fossil in the background², it produced no error messages for me, only it was listening. I hit ^C to stop it after the first page loads because the socket file is gone.

Fossil is awesome. Thank you.

[Edited to fix the fossil version number.]

1. ^{^} [faf4b4718d] was copied and pasted from the top of the timeline since I thought that was how to get the version. But I double checked the output from my fossil build and I see it's different. This is what I see at the bottom of the webpage on my repo: This page was generated in about 0.006s by Fossil 2.25 [668fefe796] 2024-08-16 19:28:37. Sorry for the mistake.
2. ^{^} Removed & from the command line.

It seems fossil deletes the socket after it’s used once.

The reason for this is clear and we'll have a fix for it soon.

The socket is deleted in the atexit() handler, which is called not only for the main parent process, but also for each forked child (a group each of the http responses belongs to).

If you would, please test this patch:

$ f diff
Index: src/cgi.c
==================================================================
--- src/cgi.c
+++ src/cgi.c
@@ -2667,10 +2667,11 @@
             nRequest++;
           }
           close(connection);
         }else{
           int nErr = 0, fd;
+          g.zSockName = 0;
           close(0);
           fd = dup(connection);
           if( fd!=0 ) nErr++;
           close(1);
           fd = dup(connection);

It looks, to my eyes, like it will resolve the issue but i'm not sure whether it will have undesired side effects.

Seems like a good patch to me.

Thank you so much Stephan! I saw the update in fossil’s timeline so I pulled it into mine and built that.

Fossil 2.25 [abeb57d28e] 2024-08-21 11:36:54

I’m very excited!

The only issue I bumped into was, I’m pretty sure, due to how I launch fossil. I don’t run it as root or under an Admin account. I open a shell as an Admin then sudo -u as the fossil-specific user who runs the fossil binary. That user has no home folder or shell. I’m guessing that’s why --socket-mode and --socket-owner threw errors.

So I removed them from my script, put in a few second sleep, then chmod the sockets.

👍🏼 Very happy! 🎉

It's ok to run Fossil as root. If you do, it will automatically

• Put itself into a chroot jail rooted at the directory that holds the repository database.

• Drop privileges - becoming the user and group that owns the repository database.

It does this prior to reading a single byte from the socket. So there is nothing the HTTP request can do to cause problems while Fossil is still running as root.

I claim it's safer to do things that way. Because by allowing Fossil to put itself into a chroot jail, if somebody does ever find a bug that allows them (for example) to shell-out, they would still be stuck inside a chroot jail and thus less likely to inflict any damage. It's a extra layer of defense. If you do not run Fossil as root, it cannot create the chroot jail, and hence you lose that extra defense.

Hm...sounds like more homework for me. :-)

One issue I ran into awhile ago is the main OS I (& family members) use makes it extremely difficult to run software as root.

Wading-in-over-my-head-warning... :-)

I thought the reason various software (code) compiled into multiple binaries, was because a) some components never needed privs beyond the current user running it, and b) an admin-level binary did a bunch of administrative setup before exiting, including launching another lesser-privileged binary as a process which was the actual (for example) Internet-facing app doing the work.

Using OpenSSH as an example, it already builds into multiple parts. Now it’s been split up even further. Although I have yet to build the latest version. Re-reading the 9.8p1 release notes, apparently even more sshd separations are planned.

Refreshing my memory, I re-read the launchd .plist I wrote for OpenSSH. I do run sshd as root:wheel at “system startup” from /Library... I also see it can be forced into a WorkingDirectory, which I ended up commenting out.

All that may be irrelevant. I don’t know.

Hm... $ man chroot makes me think I may be able to do it within the script. But I definitely like to KIS too. So I guess it’s a matter of less friction for me & my potential users: launchd fossil as root¹ or fiddle with my scripts.

Both of those are why I gravitated to fossil in the first place! :-) To keep track of what I wrote, keep track of what works, and create trials/branches to experiment. I’ve barely scratched the surface of fossil and there’s much more I need to learn.

1. ^{^} There are issues with some apps being launched this way, the OS just flat-out refuses to let some of them run via launchd, even as non-root. They can only be opened by a human user manually (by a script) after logging in. Not user-friendly.

If possible follow drh's post. If you cannot run fossil as root for some reason you have still options.

1. If fossil and the client run as the same user no further action is required.

2. If they run as different users either:

   1. Make them share a group and set the group of the socket to it
   2. Set the socket group to one of the clients groups. This is more secure.

For the latter approach create the socket in a directory owned by the fossil user and the clients group. Set the group sticky bit on the directory. Then every file (and socket) created inside the directory will be group-owned by the clients group.

mkdir path_to_socket_dir
chown fossil_user path_to_socket_dir
chgrp client_group path_to_socket_dir
chmod 2750 path_to_socket_dir
# same as:
# chmod u=rwx,g=srx,o= _path_to_socket_dir_

2 out of 3 do NOT remove the socket.

While that's true, is /var/run not a tmpfs of some type on OpenBSD?

I've been thinking on the same lines, that the value proposition of Unix domain sockets — especially as presented here in this thread — is that they have persistent permissions, and that therefore they should not be deleted between Fossil server restarts, else why speak of persistent file permissions at all?

Yet, when I went looking, I found that all of the Unix domain sockets on the nearest-to-hand Linux box are on tmpfs of one type or another:

$ sudo lsof -U | grep / | grep -c -e /run -e /proc

I get 89 here!

If you then flip that grep around, you may find that your system, like mine, has none of the other sockets in non-tmpfs directories.

This doesn't completely obviate the argument about permission persistence since the lifetime of the daemon and the uptime of the OS that hosts them may differ, giving a window where the permissions are set and left alone through the system's uptime, but it does at least undermine the argument that these sockets are better because they remember their permissions through reboots.

More and more, the only value prop I see remaining in this feature is the infinitely wider namespace. (File paths instead of 16-bit port numbers.)

Is it just a single socket per Fossil server? If so, I don't know why it should be deleted.

I would say, that if the socket was not created by fossil it should not be removed. It could get even further: delete socket only if created in default location - so only in situation where user may not know the exact location of the socket-file.

2 other common examples of unix socket usage:

OpenSSH client when using the "ControlMaster" feature (for session multiplexing)

• With "ControlMaster auto" and usually "ControlPath ~/.ssh/%C" (or similar).
• The socket is removed when the master connection is closed.

There's also tmux.

• By default, it use something like: /tmp/tmux-<uid>/default=.
• It is removed after last tmux session is closed.

Both of this examples are sockets with variable names (for single-run use), so removing them on exit is rational and desirable.

Not sure what you mean by "single run use", especially for tmux, which start a daemon and is usually use to be detached and have session running in background.

But I get what you mean and I agree. The fact that tmux use a unix socket is an implementation details so the user don’t need to know the socket even exist, so it’s better that it get deleted by tmux.

By "single run" I meant the lifespan of the daemon creating the socket.

I updated the setup to the most recent fossil version and it should work now reliably.

The initial implementation had not yet permission settings of the socket in place - that's why you got the permission denied error.

... At any rate, I'm trying to wrap my head around this particular "reverse proxy" setup. I assume NaviServer is opening up the socket and just serving whatever Fossil dishes out on that socket?

It's the other way round: fossil is creating the socket and listening on it for http requests. NaviServer (or any other reverse proxy) then connects to the socket. You must tell the reverse proxy the path to the socket.

So I still don't see a use-case for unix domain sockets in Fossil.

I am not an expert but I think in both cases 1 & 2 it’s possible to have an intermediary process on the same computer in between fossil and the HTTP requests.

Unix domain sockets, at least to my limited understanding, reduce attack surface, and definitely make scripting a connection between fossil and the in-between software, much easier than dealing with port numbers.

In fact I thought it would go into althttpd, then “automatically” end up in fossil but I see I’m wrong.

Similar request... forumpost/8f21936ebd0b76e8