You’re apparently doing this under “fossil sql”, but is it just firing up the shell that does this, or only for certain queries? What platform? For all repos or just one?

Details!

I'm with Warren on this. I could make some guesses about what is going wrong. But unless and until I can reproduce the problem, I have no way of knowing whether or not it has been fixed. Please, therefore, explain to us how you generate this error.

Here are the details:

# uname -a
Darwin 20.4.0 Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:47 PDT 2021; root:xnu-7195.101.2~1/RELEASE_X86_64 x86_64

# fossil remote-url
https://fossil-scm.org/forum

# fossil sql <<EOF 2>/dev/null
.bail on
.headers off
.mode tabs

select  config.value as value
from    config

where   config.name = 'short-project-name'

limit   1;

.exit
EOF

Does not reproduce for me. Anybody else able to generate this error?

Not for me on trunk or the commit the OP is using.

$ uname -a
Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

A bunch of us have been trying that over in chat, and none of us can reproduce your symptom. We've tried with:

• Raspberry Pi
• Linux on x86_64
• Catalina 10.15.7 on x86
• Big Sur 11.3.1 on both M1 and x86_64
• Fossil release 2.15.1 (src:2f901f98b3)
• Fossil tip-of-trunk

Furthermore, your test seems to be overly complicated for what you're trying to achieve, which makes me wonder if your actual symptom comes from some quite different purpose, which you're hiding. This gives the same result without nearly as much code:

   $ fossil sql -R ~/path/to/fossil-forum.fossil \
     ".mode tabs" \
     "select value from config where name='short-project-name'"

That is, you don't need…

1. …the .bail on bit at all, as far as I can tell. This query returns a string or nothing. Why do you need bail-on-error behavior?

2. …the .headers off bit: that's default behavior.

3. …the LIMIT 1, since there is only ever zero or one name-to-value mapping in this table.

4. …the over-specified config table names. There is no ambiguity requiring that here.

5. …the redundant .exit at the end: it's going to exit regardless.

Maybe explaining what it is you're really trying to do would help us reproduce the symptom?

Further details...

(1) The 'fossil sql` command is part of a larger script processing a given artifact — using various fossil cli commands.

(2) Several of these scripts run concurrently (~ a small dozen) over distinct artifacts.

The issue arises intermittently when multiple scripts run in parallel.

What else would help to narrow this down?

Thank you for the follow-up.

Yes, this is part of a larger script, which uses various fossil cli commands.

That said, this is the only use of fossil sql — as the config data is not exposed through the regular cli as far as I know.

When the script is run serially, processing one artifact at the time, there is no apparent issue.

When several instances of the script are run concurrently, the assertion fails every now and then.

Stylistic issue aside, what else would help narrow down the issue?

Thanks for the help!

What else would help to narrow this down?

An actual reproducing test case.

A reproducible bug is a dead bug walking.

An intermittent bug relying on underspecified behavior rarely encountered in the wild is likely to live a very long time.

Does the symptom recur if you rewrite the code as I've shown, or does it require each-and-every one of the oddities I've flagged to show up?

For instance, if removing the redundant .exit fixes it, then:

1. You have an immediate workaround while we're working on the actual problem; and

2. We'd then know there's something about the double-exit that triggers the problem.

If the symptom entirely goes away when rewriting the query as I've shown, then the next step would be to bisect the difference between the two: reintroduce one of the flagged oddities at a time until the symptom comes back, at which point you have the likely culprit.

How is the concurrency be done?

background process (i.e. #script &)

Please rebuild using check-in eddfa8dfbe830c27 or later and let us know whether or not it clears your problem. Thanks.

There's a secondary symptom besides the debug assertion, where doing too many "fossil sql" calls in parallel can result in a "database is locked" error. This is because "fossil sql" doesn't retry the DB conn if it's busy or locked, doubtless for good reasons, but until there's a way to make it do so, you may want to patch your local Fossil like this:

Index: src/shell.c
==================================================================
--- src/shell.c
+++ src/shell.c
@@ -14757,12 +14757,20 @@
             SQLITE_OPEN_READONLY|p->openFlags, 0);
         break;
       }
       case SHELL_OPEN_UNSPEC:
       case SHELL_OPEN_NORMAL: {
-        sqlite3_open_v2(p->zDbFilename, &p->db,
-           SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE|p->openFlags, 0);
+        int i;
+        for (i = 0; i < 15; ++i) {
+          int rc = sqlite3_open_v2(p->zDbFilename, &p->db,
+             SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE|p->openFlags, 0);
+          if( rc==SQLITE_BUSY || rc==SQLITE_LOCKED ) {
+            sqlite3_sleep(1000);
+          }else{
+            break;
+          }
+        }
         break;
       }
     }
     globalDb = p->db;
     if( p->db==0 || SQLITE_OK!=sqlite3_errcode(p->db) ){

That allows $CORE_COUNT instances of back-to-back "fossil sql" calls in parallel on a 4-core Hyperthreaded box to run without ever getting locked up. It's possible to run out this hard-coded 15-tries-once-a-second retry loop by doing even more parallel runs per second, but if you've got that much overlapping concurrency going on, you'd probably do better to throttle calls to a reasonable number rather than increase this timeout further. Running 1000 parallel queries on an 8-core box won't make it go faster than running 8-16 or so.

This problem is fixed a different way on trunk now.

This is because "fossil sql" doesn't retry the DB conn if it's busy or locked

Oh. Good point. Thanks for pointing this out.

Is fossil sql the only cli command exhibiting this behavior at the moment?

Either way, thank you very much for the patches; will give them a spin and keep on monitoring the system behavior.

Thanks again for all the help.

The commit linked from the prior post changes that. Now it retries for up to 10 seconds before giving up.