"Debian" isn't enough for me to replicate your symptom. My local Buster VM doesn't even allow static linkage due to changes in glibc, so you must be on a materially different version of Debian.

I've attempted to address this issue on the new allow-static-openssl branch. Please give it a try. I won't merge it to trunk until I can prove that it actually produces a runnable build, which I can't do locally given my local set of test VMs.

Alternately, provide enough info that would allow me to build a VM matching your environment:

 $ uname -a
 $ lsb_release -a

etc.

"Debian" isn't enough for me to replicate your symptom.

Of course. Sorry. Since I had succeeded the staticatically linked compilation by modifying the Makefile, my intention was just to know if someone had had a similar issue, before investigating further.

I compiled Fossil 2.14-preview-20201125 on an updated Debian 10.7 "buster", 64-bit Intel.

uname -a
Linux 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux

I've attempted to address this issue on the new allow-static-openssl branch. Please give it a try

Thank you. Now configure --static adds -lpthread to LIB in the Makefile:

Before (v2.14-preview-20201125):

LIB =    -static -lfuse -lm -lresolv -lssl -lcrypto -lz -ldl

Now:

LIB =    -static -lssl -lcrypto -lpthread -lfuse -lm -lresolv -lssl -lcrypto -lz -ldl

But the compilation fails as before, after displaying many "pthreads_" undefined references, and some "sem_".

When I move -lpthread to the end of the list, it works.

drh's recent change on trunk should work for you, then.

Now ./configure --static creates a Makefile with this line:

LIB =    -static -lpthread -lm -lresolv -lssl -lcrypto -lz -ldl

But the compilation fails after displaying 11 messages like this, about different "pthread_" undefined references in different functions of the same library:

/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libcrypto.a(threads_pthread.o): in function `CRYPTO_THREAD_get_local':
(.text+0x143): undefined reference to `pthread_getspecific'

Again, when I move -lpthread to the end of the line, make clean;make works.

I've made another change to it which should solve the problem.

Yes, now it works without touching the Makefile. Thank you!

Fossil itself does not use pthreads. It is single-threaded. (The SQLite build contains SQLITE_THREADSAFE=0 so that SQLite does not attempt to use pthreads either).

I'm guessing that the pthread requirement comes about because you are linking against OpenSSL.

Almost certainly, which is why my branch only affects proc check-for-openssl in auto.def.

Fossil ... is single-threaded

This is surprising. Especially within the context of a long running SCGI server.

Please provide some documentation of both threading model and database-locking model of Fossil.

1. What happens on simultaneous HTTP requests? Does it fork?
   
2. What if a CLI is invoked during HTTP request (or vice-versa)?
   
3. How a program can safely and efficiently read and write a repository that is being served with fossil server?
   
4. What if this sever is under high load?
   
5. What if this program is invoked through CGI extention?
   
6. Are there any special recommendations for the Wapp scripts?
7. What is the recommended connection string for Go programs?

Fossil uses a process model. Instead of starting a new thread, it starts a new process. A process is, after all, just a thread with its own private address space. A process better than a thread, because a oricess has its own address space and hence there are never any concerns about concurrency control.

1. Every HTTP request is handled by a separate process. Also every SCGI request. The fork() system call is very fast (on Linux).
2. The CLI is obviously running in a separate process from any servers that happen to be on the same system.
3. Multiple processes can read/write the repository as SQLite handles concurrent access to the repository database automatically. Fossil doesn't have to worry with that.
4. Server load has nothing to do with it. Every request is a separate process.
5. Every CGI request is a separate process. This is how CGI works. This is true of all CGI programs, not just Fossil.
6. Wapp is also single-threaded.
7. I don't have any idea how to do concurrency in Go.

Perhaps you've been told that multi-threading is necessary for efficiency. Fossil stands as a proof by example that this thesis is wrong. It is quite possible, and in fact not all that difficult, to be lightweight, efficient, and responsive, while still delegating each HTTP/SCGI/CGI request to a separate process. Remember, Fossil runs easily on a $5/month VPS or on a RaspberryPi. Competing multi-threaded systems (ex: GitLab) require an order of magnitude more processing power, or so I'm told.

Adding to drh's comments, point-by-point:

1. On Windows, it's complicated. The initial HTTP request is handled by a thread, but it writes info from the HTTP request out to temporary files and passes them to a subprocess which reads them back in. Other commands that do fork()-like things on Linux also use processes instead in native Windows builds of Fossil, though with stdin/stdout tied to the parent via Win32 pipes instead of using temporary files.

   It'd be interesting to benchmark server operations of a native Windows Fossil server versus operation on the same hardware under WSL2.

   WSL1 and Cygwin should be slower than either, but I'd encourage someone sufficiently interested to benchmark them, too.

2. SQLite transactions arbitrate access. Under some conditions, you can get a lock contention that requires one of the two processes to retry. This usually indicates that you're trying to do something like fossil serve on the same repo DB file you're committing directly to, which you can fix by committing to a clone rather than to the same DB file, so that arbitration goes through the sync protocol instead of thru the lower-level SQLite layer.

3. Ditto.

4. Define "high." Is it higher than sqlite.org, for which we have an existence proof of adequacy? It achieves that in part by using features documented in "Managing Server Load".

   If instead we're talking about something far less busy than sqlite.org — which I'd guess is almost all uses of Fossil — then the only way I see that you can get into trouble short of a DDoS attack is underprovisioning your server. You might need a $10/mo VPS instead of a $5/mo VPS, for example.

5. Then you have a process-per-conn model, so see point #1. (i.e. How many processes fit into RAM under fork() vs re-exec models based on host OS, etc.)

6. No idea.

7. I'm reading an implication that you're doing SQL queries against the Fossil repo being served, which begs many new questions. If so, please justify those uses versus, say, "fossil sql" commands, which delegates such details back to Fossil, which we may presume does things correctly.

Richard and Warren, thank you for the explanations!

Define "high"

In my brave dreams that's on the order of a thousand users that actively read (on order of 5 requests per minute per user on average) and occasionally write (on order of 5 requests per hour per user on average).

doing SQL queries against the Fossil repo being served

Yes. Thanks you sagacity.

please justify

Basically three reasons:

• ability to insert data into fx_* tables (and use it from within the Fossil)
• ability to scale-up the rate of write-requests from these hmm... let me name it sensors.
• it just "feels wrong" to finagle with formatting/IO/parsing when a direct API is (seemingly) available.

lock contention

Yes, that was on my mind. I thought that the OS's kernel can manage in-process spin-locks, mutexes and semaphores much faster than filesystem locks. But I may be wrong on that. In no way I'm a kernel guru.

What I'm actually more anxious about is stability/correctness. Sqlite3 is a Magic Wand that Just Works, but one still have to know "the right spell" (e.g. database opening parameters).

fossil sql

Yes, I intend to employ native Fossil commands whenever there is a risk to mess up the repository.

UPDATE: I'll try to learn from The Guru :-)

5 [read] requests per minute per user…5 [write] requests per hour per user…

We should probably take that up on the SQLite forum, but briefly, you should be okay. The main difficulty is SQLite's strong locking and single-threaded nature, which limits you to ~60 TPS on a fast spinning disk, but orders of magnitude more on an SSD.

Modern SSDs can be made strong enough to withstand such TPS rates through the expected life of the drive.

ability to scale-up the rate of write-requests

What sort of scaling are you talking about?

There's vertical scaling, where the hardware's the main limit on TPS rates and such. This is what I've been talking about so far.

There's also horizontal scaling, but I don't see how it would apply here. That would imply that you've got a cluster of Fossil servers and are somehow syncing them, but that doesn't happen with fx tables and other things you're talking about.

a direct API is (seemingly) available.

Are you speaking of this?

I thought that the OS's kernel can manage in-process spin-locks, mutexes and semaphores much faster than filesystem locks.

I don't see how you can avoid filesystem locks for an ACID-compliant DB stored on a filesystem. See this doc for details on SQLite's approach.

Ignore the big red hints to go read the WAL docs instead. Fossil doesn't enable WAL by default, so it's unlikely the WAL docs apply to your use cases unless you go out of your way to enable it.

I asked why we don't use WAL by default once, but got no answer, so here's me asking again... :)