How deprecated is using an external SQLite for Fossil? Due to a chain of dependencies, this matters to LumoSQL and I'm hoping the answer is this is definitely deprecated and we should say so in the docs.

It's clearly somewhat deprecated, and as drh and others pointed out here it implies an unsustainable support burden on both Fossil and SQLite. There is also the endosymbiotic point as illustrated by recursive common table expressions (CTEs), meaning Fossil needs SQLite-latest regardless of the environment Fossil is deployed into.

Finally and perhaps most damningly, there is the liklihood of Fossil being unusable in common distributions in the future. At least three Linux distributions (Gentoo, Debian and Ubuntu) currently build using the Fossil configure option --disable-internal-sqlite .

Can I modify the docs and maybe configure to say "don't use --disable-internal-sqlite"?

Dan Shearer

My answer to this is: Please, No. If 4 distributions (Gentoo, Debian, Ubuntu and count Cygwin too) use --disable-internal-sqlite) it's for a reason. Yes it makes it more difficult that no-one succeeded to convince dhr how useful it is, I don't think it's possible to clear up this dispute (there are stubborn people in the world, I'm one of them ....).

Jan Nijtmans (jan.nijtmans) wrote on 2021-02-25 12:12:32:

Yes it makes it more difficult that no-one succeeded to convince dhr how useful it is

Your point is that this is so useful that it needs to be possible, and that you accept the risks involved. That means "don't deprecate".

drh's point is that this will introduce errors and unintended consequences, and these errors won't always be obvious by looking at the data afterwards. That means "warn users strongly".

Proposed Action 1 Fossil should always issue a warning if it detects that it is using an external sqlite

Proposed Action 2 Rename --disable-internal-sqlite to --risky-unguaranteed-disable-internal-sqlite

Including a library is called "vendoring" as I explained in the Not-Forking README and is usually a bad thing. That's why the distributions are splitting it out. But this is a highly unusual case of vendoring:

1. The library (SQLite) and application (Fossil) are maintained by overlapping teams, with the same principal developer
2. The application release schedule is often driven by the library release schedule
3. The library release schedule could well be driven by the application release schedule as drh explained

Proposed Action 3 Distribution packagers should be pointed at this discussion and asked if they can please reconsider splitting out sqlite

Dan Shearer

this will introduce errors and unintended consequences

The general rule for package maintainers: "Don't upgrade fossil without upgrading SQLite too". That's why fossil checks for the minimum SQLite version it works with, and the minimum SQLite features that must be enabled, already. That warning should be sufficient in 99% of cases. The remaining 1% happens when Fossil is released with an unreleased alpha of SQLite included. No problem for the Fossil/SQLite development team, but that means extra work (or delay) for distribution maintainers. No problem for me: I know what I'm doing ;-)

Proposed Action 3 Distribution packagers should be pointed at this discussion and asked if they can please reconsider splitting out sqlite

Feel free to do that, but I already know their answer .....

Distribution packagers should be pointed at this discussion and asked if they can please reconsider splitting out sqlite

I think you would sooner convince them to convert from vi to emacs.

I think the separable SQLite thing has to stay, regardless of how much I dislike it.

The general rule for package maintainers: "Don't upgrade fossil without upgrading SQLite too".

FWIW, i'm 100% behind Dan's proposal. The only reasonable alternative would seem to be to hold package maintainers to what Jan says above.

i also want to point out that the sqlite devs have long suggested that apps embed their own copies of sqlite, and it is in no way uncommon. Certainly many apps which do so do not offer the option to switch to a system-level copy. Perhaps we just need to remove that flag altogether - problem solved.

As I am seemingly late to the party: What is the reason the package maintainers (as I get it, their being the main party stoppers) prefer to use the system sqlite library? Since Fossil contains the sqlite sources, why would one be against letting it use them?

What would happen if we just remove the option of using the system library?

Many distros have a "no static linking" policy. They want all libraries to be shared libraries. That way, if a security vulnerability is discovered in a library, they can fix it simply by updating the library and not having to also update every application that links against that library. (In theory, at least.)

Distros tend to be very dogmatic about this policy. No exceptions. Ever. The matter is not open for discussion or reasoning.

Daniel Dumitriu (danield) wrote on 2021-02-25 13:32:04:

What is the reason the package maintainers (as I get it, their being the main party stoppers) prefer to use the system sqlite library?

Because that is generally the correct behaviour. Search for all occurrences of the word "vendoring" in the Not-Forking README for an explanation, links and exceptions.

What would happen if we just remove the option of using the system library?

Like, say Firefox, as Stephan pointed out above? It would cause awkward discussion, as a minimum. And quite likely people maintaining the very simple patch set required to put it back in. Suddenly removing an option is not right, which is why I suggested renaming it.

However, unlike Richard above, I do think there is a sensible discussion to be had. This is a very unusual case, and I really don't think it can be classed as vendoring in the way that causes distributions grief. The distribution maintainers are not crazy people, what they want to avoid is potentially risky proliferation of libraries that each will need updating in a heartbleed event, and which can't be modified to distribution-standard in one place (eg with particular optimisations switched on.)

Dan

Richard Hipp (drh) wrote on 2021-02-25 13:55:01:

No exceptions. Ever. The matter is not open for discussion or reasoning.

You are referring to examples like the iproute2/libbpf debate .

I disagree with your conclusion that there is zero room for debate. I think Fossil/SQLite is much closer to the case of Kubernetes in Debian:

The Committee declines to overrule the maintainer and accepts the level of vendoring used in the 'kubernetes' source package. We also decline to intervene in bugs requesting that vendored components in the 'kubernetes' source package be installed in binary packages in such a way that other packages can make use of them.

Our consensus is that Kubernetes ought to be considered special in the same way that Firefox is considered special -- we treat the package differently from most other source packages because (i) it is very large and complex, and (ii) upstream has significantly more resources to keep all those moving parts up-to-date than Debian does.

Fossil/SQLite is much less complex, but also fast-moving. And Fossil/SQLite have a large overlap in maintainers and authors which Kubernetes doesn't have.

Dan Shearer

Jan Nijtmans (jan.nijtmans) wrote on 2021-02-25 13:08:05:

The general rule for package maintainers: "Don't upgrade fossil without upgrading SQLite too".

How can that be? A distribution might have dozens to hundreds of packages that dynamically load SQLite. SQLite does try to avoid surprises, but upgrading them all frequently just because one application (Fossil) is fast-moving doesn't seem a good packaging strategy.

Dan Shearer

I think the distros solve this by simply not updating Fossil very often.

The current solution to this problem has been working well. Distros can ship older versions of Fossil and SQLite. People who want the very latest stuff (ex: /chat) can download statically linked binaries to drop onto their $PATH. Everybody is happy this way. Everybody can get what they want/need.

Why do you want to disrupt the system that has worked so well for so long? What problem does it cause you that you are trying to solve?

There are two very different classes of package systems.

1. Rolling releases, where packages get updated as fast as the package maintainers wish to push them out. For these, there are good reasons to choose an external SQLite: efficiency, security, and overall simplicity, sub-optimizing the parts to optimize the whole. (Smaller download sizes, external sqlite3 that matches fossil sql, etc.)

2. Stable platform distros, where they pin core packages like SQLite in place for years, which then sets the version of peripheral packages they can support atop that base. This is what drh is talking about above. It's why Debian 9 shipped Fossil 1.37 through its entire lifetime and Debian 10 still ships Fossil 2.8 despite the OS being nearly 2 years old now. Debian 11 is likely to ship 2.14, which will then be pinned in place for another 2-3 years.

The second option can be highly-secure as well, provided security fixes are quickly backported. This is why Red Hat is a multibillion dollar company: it takes a lot of effort to do that, so it's worth paying for if you want both stability and security. Such an organization is going to prefer building their updated libsqlite3 package once and push that out rather than rebuild everything that statically builds SQLite in, then push all of those out.

These are both sensible design choices. Fossil should support both. We can't wag the dog on this one.

Richard Hipp (drh) wrote on 2021-02-25 14:36:12:

Why do you want to disrupt the system that has worked so well for so long?

You don't like it, but acknowledge it has been working well. Ok I hadn't noticed that distinction. I thought you felt it was a problem to be solved.

What problem does it cause you that you are trying to solve?

It's a chain of causality documented in this ticket.

The sentence in that ticket that led me to starting this thread is:

Given that Fossil should always be built using a current SQLite we don't need to be worried about an old system SQLite for Fossil. At least if external sqlite is deprecated in Fossil.

... and then I started wondering. How true is this? And the more I looked, the less true it was, despite Fossil/SQLite developers saying forcefully that it should be true.

So I used the knowledge of vendoring I got from Not-Forking to suggest that there is a case for the distributions being happy to package SQLite-latest when they package Fossil.

Dan Shearer

Package Maintainer's Response

I spoke with Barak A. Pearlmutter, the maintainer for Fossil in Debian. Barak said I could quote as follows:

Right now, fossil contains a copy of sqlite which can be enabled/disabled at build time. The Debian package used to simply disable the internal copy, in favor of the system sqlite library. But not too long ago, there was an issue where the system sqlite was too old: fossil used newer features that were in its internal sqlite but not in the system library. So I hot-wired the Debian build scripts to check if the system sqlite library was recent enough, and use it iff it was. This makes the build process more robust, especially for things like back-ports. It also handles the case where the system sqlite library is not available at all, which is nice for building fossil on embedded devices and such.

So it seems that at least in Debian there is a good understanding of this issue. The point about backports is particularly relevant, because Fossil is going to need the internal build pretty much every time.

Package Maintainer's Request

Barak has a request for the Fossil dev team:

Of course, my hot-wiring was pretty crude. Making it easier to do this sort of checking might allay some people's concerns about vendoring.

Right now, when there's a new release, I do

$ egrep -i 'sqlite.*at least' src/main.c fossil_panic("Unsuitable SQLite version %s, must be at least 3.34.0",

and then hard-code that in debian/rules with

SQLITE_MIN_VERSION=3.34

which is manual and hideous. If there were some supported way to get the minimum required version, that would be nice. Especially since it sounds like it's going to be bumped again soon.

To me this suggests a configure option to emit a minimum sqlite version for consumption by packaging-specific scripts.

At least this would reduce the library-splitting and it would reduce the chances of compatibility problems. This is a start.

Dan Shearer

Maybe an option to configure: --prefer-system-sqlite?

This would link to the system sqlite if the system sqlite is newer than the minimum required version. If the system sqlite is too old, it will use the in-tree version.

This would also require that the build system is the same as the target system. So it wouldn't work for cross compiling or trying to run the binary on an earlier OS (assuming it would work). (However the version check inside fossil should prevent it from actualy running on a system where the instaled sqlite is less than the minimum version.)

So I am not sure if this magic will cause more issues than it solves but it is a better option than manually setting SQLITE_MIN_VERSION.

... But not too long ago, there was an issue where the system sqlite was too old: fossil used newer features that were in its internal sqlite but not in the system library....

It would be nice if internally Fossil had some way of cataloging which SQLite's features are required for a given Fossil operation. Thus the version compatibility could be checked on the Fossil feature level.

So a user with a lower version of SQLite library may not be affected at all, if using just a subset of features, and is given a warning/error when attempting to use the features unsupported by their SQLite version. Basically, a graceful degradation.

I understand that it should always be possible for a user to install the up-to-date "vendored" version of Fossil locally, if desired.

It would be nice if internally Fossil had some way of cataloging which SQLite's features are required for a given Fossil operation.

There's a really simple answer to that: fossil and sqlite are developed hand-in-hand, literally at the same keyboard/chair combination, and most cutting-edge features are tried out somewhere in fossil at some point. Every time a new feature or significant bugfix lands in the sqlite trunk, it's typically in fossil's trunk within a matter of hours.

i.e. it always needs the latest feature(s) from the latest version (often a pre-release). A version check for sqlite is superfluous.

Edit: okay, that's slightly exaggerated, but not by terribly much.

...and most cutting-edge features are tried out somewhere in fossil at some point.

That's one more reason to have such a feature-test in place before the use, instead of a blanket MIN version fail at start.

Most of Fossil's core functions are based on very much long-existing and equally core features of SQLite. When a new SQLite feature is added to be used "somewhere" in Fossil, it's usually tied to some new feature in Fossil. So the feature-test and warning/degradation can just be placed specifically there, on that new Fossil feature.

It appears that the mandate to use the embedded SQLite version is what should called superfluous, at least as of present, when Fossil has a formalized use as an individual SCM tool and thus a set of core features for such use.

After all, Fossil is not packaged into distributions as a "testing ground for SQLite development" but rather as a general SCM tool.

So a decoupling is reasonable and SQLite feature-use inventory (at least going forward from the present state) is one way to accomplish such decoupling.

Most of Fossil's core functions are based on very much long-existing and equally core features of SQLite. When a new SQLite feature is added to be used "somewhere" in Fossil, it's usually tied to some new feature in Fossil. So the feature-test and warning/degradation can just be placed specifically there, on that new Fossil feature.

And when a feature is extended/remade (which is the most common situation in fossil development), but does not use a brand new sqlite capability, then who should check when newly used capabilities have appeared in sqlite? Are you saying, that it would be better to waste developers time on sqlite archeology, just to help some users use a broken system (one with unmatched sqlite and fossil versions)? Isn't it easier for all to just send them to the download page, where they can get a prebuild binary or get the sources and build them for they own use?

Many distros have a "no static linking" policy. They want all libraries to be shared libraries. Is there a reason that Fossil+SQLite releases cannot be reasonably coordinated. Both of these projects are definitely under direct management. So this could be just a matter of some policy/convention/automation to be in place such that the maintainers would grab the mutually consistent releases of Fossil and SQLite.

This way, the SQLite source version to be included in the distro would be the very same version the Fossil source would internally use as well. So that this could equally allow building Fossil to link against the external SQLite library. Same SQLite version, so should be no problem either way. No vendoring, just coordination.

For example, looking at Debian Bullseye sources (as of 2021-02-27):

• fossil-2.14 (embeds sqlite-3.35.0)
• sqlite-3.34.1

As of 2021-02-27 Fossil and SQLite websites listed for download:

• Fossil current tagged source release: 2.14 (2021-01-20, sqlite-3.35.0), 2.13 (2020-11-01, sqlite-3.34.0),
• SQLite current source release: (2021-01-20, 3.34.1)

In Debian Buster: fossil-2.8 (2019-02-20, sqlite-3.27.1), sqlite-3.27.2 (2019-02-25)

Obviously, there's a mismatch of the SQLite versions.

Yet, in Debian Stretch: fossil-1.37 (2017-01-16, sqlite-3.16.2), sqlite-3.16.2 (2017-01-06)

Not sure how this translates to other Linux distributions. Perhaps, there could be a way to keep the Fossil and SQLite releases to be matching in recency at least at the level of the most current tagged release available at the download pages. This may make the maintainers' choices more predictable and could just as well solve the external SQLite library version mismatch at the distribution level, assuming the users indeed update from the official distribution repos.

This way, the SQLite source version to be included in the distro would be the very same version the Fossil source would internally use as well

Fossil is just one of dozens, if not hundreds, of packages on the affected distributions which use sqlite. If the package maintainers blindly prefer the version preferred by fossil, they risk compatibility issues with many other packages.

Just to clarify my original post (somehow the quoted text got fused with the beginning of my text). Here's the beginning text repeated:

Is there a reason that Fossil and SQLite releases cannot be reasonably coordinated?

Both of these projects are definitely under the direct management. So this could be just a matter of some policy/convention/automation to be in place such that the maintainers would grab the mutually consistent releases of Fossil and SQLite.

To address Stephan's reply: The general expectation is to have the latest Fossil release version internally use the latest SQLite release version. The SQLite in a way is a "master" here, Fossil is a "slave". Just there should be a simple convention that the current released SQLite source version is what gets included in the current released Fossil source version.

This does not preclude the interim Fossil "preview" (with whatever source version of SQLite). It would make sense to mark such Fossil versions appropriately in order to denote the internal version of the SQLite used.

For example, looking at Debian Bullseye sources (as of 2021-02-27):

• fossil-2.14 (embeds sqlite-3.35.0)
• sqlite-3.34.1

Not really a mismatch. The Fossil 2.14 release notes say: "The built-in SQLite is updated to version 3.35.0 alpha containing performance optimizations, especially performance associated with startup, and minor improvements to the CLI.". I read this as: "If you downgrade to SQLite 3.34.x, you will lack the performance optimizations and the CLI improvements". SQLite's "CLI improvements" are available through the "fossil sql" command, not really crucial for Fossil operation. My guess is that the Debian people read the changelog, concluding it's OK to use SQLite 3.34.x too. They - most likely - built fossil with downgraded SQLite, ran the testcases, and found that it worked OK. Apparently they had sufficient confidence that the combination worked, and - indeed - I think it does.

In Debian Buster: fossil-2.8 (2019-02-20, sqlite-3.27.1), sqlite-3.27.2 (2019-02-25)

Obviously, there's a mismatch of the SQLite versions.

This is not a mismatch either: sqlite-3.27.2 is a bugfix release, which is upwards compatible with sqlite-3.27.1. Upgrading should be no problem at all.

What's important here: the Fossil changelog should clearly indicate when a Fossil release uses a non-released SQLite version, and the reason why, so distribution maintainers can make a valid decision. The Fossil 2.14 release notes do that, so IMHO it's OK. If Fossil distributes a released versions of SQLite, a reference to the SQLite release notes is sufficient.

Thanks for the good work!

So, I agree with @dhr: No need to change anything, Fossil's current release procedure works fine. No complaints from me.

Is there a reason that Fossil and SQLite releases cannot be reasonably coordinated?

It simply takes effort to do that (additional SQLite release when a specific Fossil release wants a feature from it) or delay (delaying a Fossil release until the next SQLite release is out). If a Fossil feature is sufficiently important, that might be enough reason to 'violate' the requested coordination. As long as the Fossil release notes clearly indicate that, it's fine with me: So I can decide if I want to build the latest SQLite too, or simply wait.

SQLite 3.35.0 changes the meaning of the SQLITE_DBCONFIG_ENABLE_TRIGGER option slightly. The most recent Fossil code uses that option in a way that will not work correctly with older versions of SQLite. So SQLite 3.35.0 really is needed for the latest Fossil trunk. You can link the latest Fossil against SQLite 3.34.0 and it will appear to work, mostly. However, email notifications won't occur as expected.

Good to know, that's how it works, no problem at all.

The interesting thing is: would Fossil 2.14 as released work fine with SQLite 3.35.0 when it will be released? Or - in other words - do older fossil versions use the SQLITE_DBCONFIG_ENABLE_TRIGGER option in a way that would break when upgrading SQLite to the latest version?

Answering my own question: Fossil 2.14 didn't use SQLITE_DBCONFIG_ENABLE_TRIGGER at all, so it looks like Fossil 2.14 will work fine with SQLite 3.35.0 final.

Correct.

Details: Fossil 2.15 includes additional defenses against mischief by disabling triggers in the repository schema. An attacker might add triggers to a repository then somehow convince an unsuspecting user to run Fossil commands against that repository. Perhaps some weakness could be found that would cause the malicious trigger to cause harm. I don't know how a malicious trigger might cause harm. As far as I am aware, adding arbitrary triggers to a repository schema would do nothing worse that corrupt the data in the repository, but anybody who can insert a malicious trigger can do that without the help of the trigger. So disabling triggers does not close any known holes. But it does provide an extra layer of defense against zero-days.

However, Fossil uses triggers as part of email notification. To make this work, I have now converted the email notification triggers to be TEMP triggers that are inserted as needed, on each database connection separately. The SQLITE_DBCONFIG_ENABLE_TRIGGER flag is modified to only affect triggers in the repository schema, and allow TEMP triggers through, since an attacker has no ability to insert TEMP triggers.

Fossil 2.14 works with SQLite 3.35.0 because Fossil 2.14 and before do not make use of SQLITE_DBCONFIG_ENABLE_TRIGGER. Fossil 2.15 does not work with SQLite 3.34.0 however, because Fossil 2.15 does use SQLITE_DBCONFIG_ENABLE_TRIGGER but expects it to only affect schema triggers and leave TEMP triggers alone, which is new behavior in SQLite 3.35.0 and later.

Proposed Immediate Action

Good discussion. I now propose adding a file called MINIMUM-REQUIRED-SQLITE-VERSION to the root directory of the Fossil repo, containing only a string representation of an SQLite version number, eg "3.35.0".

This is the minimum change to both make Packagers happier and improve things long-term for the rest of us.

Ok if I do that?

Summary of Discussion

It's worth reading the whole thread, including comments by the Debian Fossil packager and multiple Fossil developers. It became clear that Fossil developers were unaware that packagers already use the internal SQLite when they think it matters, and, that Fossil packagers have no good way of deciding when it matters because Fossil does not currently communicate this.

This feels more like a useful meeting of minds and less of a conflict of goals than previously suggested.

A current example came from drh:

SQLite 3.35.0 changes the meaning of the SQLITE_DBCONFIG_ENABLE_TRIGGER option slightly. The most recent Fossil code uses that option in a way that will not work correctly with older versions of SQLite. So SQLite 3.35.0 really is needed for the latest Fossil trunk. You can link the latest Fossil against SQLite 3.34.0 and it will appear to work, mostly. However, email notifications won't occur as expected.

Dan Shearer

A file name that long will make ls output take too many lines, and it may clip in some GUI file browsers.

It’s also redundant: we already document this in the change log.

Rather than add a second place to find the info, let’s just formalize its presentation so it can be reliably extracted:

  $ ggrep -m1 'Minimum required SQLite version: [0-9.]+$' www/changes.wiki | grep -o '[0-9.]+'

On a system without GNU grep, replace the -m1 bit with a pipe thru “head -1”.

Then, document the required format in the release doc.

Rather than add a second place to find the info, let’s just formalize its presentation so it can be reliably extracted:

If it were in the configure script instead, it would be enforced instead of advisory.

What, automatically fall back? Maybe allow both directions: prefer built-in or prefer external, but only if both fail, fail the build?

That should work.

What, automatically fall back? Maybe allow both directions: prefer built-in or prefer external, but only if both fail, fail the build?

i was only thinking as an alternative to the change log, but... the idea certainly has some appeal.

Or use BUILD.txt for the same.

Is providing a way to query this from/with the executable a totally bad idea? Maybe as an extra in fossil version -v? Or some other flag?

That creates a chicken-and-egg problem: you have to build Fossil to learn what Fossil requires.

Keep in mind that in most packaging systems, you're building within some sort of temporary container that's destroyed after the package builds successfully. The host of that container almost certainly won't have Fossil on it at all, much less place it where the package build process can find it.

Even if you got around all of that somehow, you'd have to arrange to run the same version of Fossil as you're trying to build.

The only way I see out of that trap is to first build a version statically linked against the included SQLite, then try to set that aside and build a new version dynamically linked against the platform version.

For a solution meant to be simple, it ends up being rather complex.

After reading it, it is of course obvious :)

Well, I guess a line in changes.wiki or BUILD.txt should do it.

Daniel Dumitriu (danield) wrote on 2021-03-04 13:45:22:

Is providing a way to query this from/with the executable a totally bad idea? Maybe as an extra in fossil version -v? Or some other flag?

There are already scripts run before release as specified in release-checklist.wiki.

Regardless of code being run at release, I will reflect whatever decision we come to here (and it feels close) in release-checklist.wiki .

Dan Shearer

Stephan+Warren :-) said:

If it were in the configure script instead, it would be enforced instead of advisory. Maybe allow both directions: prefer built-in or prefer external, but only if both fail, fail the build?

Yes, that makes sense.

I went looking for where to do this in the (auto)build process, and I discovered another justification for this discussion, and also I feel quite late to the party. I was surprised to discover we're already maintaining SQLite capability information in C code, and it is already out of date according to drh.

So in fact the changes.wiki suggestion would still be a second place, and we'd have to decide which is canonical, if either. For Warren's reasons, I don't think having it in two places is a good idea, and therefore I downvote the suggestions of parsing changes.wiki or BUILD.txt.

I propose we improve the following to embed a literal version number and some logic around that.

Currently in auto.def we have:

find_system_sqlite

  proc test_system_sqlite {} {
    # Check compatibility of the system SQLite library by running the sqlcompttest.c
    # program in the source tree

The file src/sqlcompattest.c appears to be testing system SQLite capability although that is in this case solely determined by version as the last commit comment by drh shows. Presumably the idea of testing capability is if the system sqlite was somehow built in an unexpected way and was less capable than the version implied.

Therefore the fix would be something like:

• add a literal minimum required version number to auto.def
• if the new build parameter --use-system-sqlite is passed, then check the found system sqlite against this version number and fail if not equal or greater than the minimum
• if --use-system-sqlite is passed, and if it is equal or greater than the minimum, then run the capability test code.

How about this?

Dan Shearer

Sounds sensible, except to repeat that you need to add this requirement that the SQLite minimum version number in auto.def be added to the release checklist.

I suppose it goes around step 14. I have no particularly strong feeling about whether it goes before or after.

State of Distributions

I asked the Fossil maintainers in four of the main distros (themselves upstreams of other distros and spins). I pointed them all at the summary of stricter version checking above.

It seems we have a happy conclusion. All four are happy to use internal SQLite, some of them with the caveat that they will use the system sqlite if it passes our tests.

• Gentoo are happy to publish Fossil current even when that means using the internal library and even if in practice this is always the case. Aaron W. Swenson adds "--prefer-system-sqlite starts to get into automagic dependency territory, and we're firmly against it"
• Debian is happy to retry builds with the internal sqlite if configure fails the system sqlite. Barak A. Pearlmutter thinks --prefer-system-sqlite is a "great idea" and says "That would simplify things for sure, not just for me but in general
• FreeBSD already defaults to the internal library, and Pietro Cerutti is happy to say that will continue
• The Fedora maintainers said thanks for the heads up, and raised no problem with the stricter tests

We can discuss --prefer-system-sqlite in the next release cycle, but this much is a good step forward.

Dan Shearer

Could the default build of SQLite be changed such that -DSQLITE_JSON1 is used?

That makes it more likely that the default build for SQLite is suitable for Fossil.

On Cygwin, using the Fossil's internal SQLite is a no-go. For various reasons. The current vfs for Cygwin is crippled. For example, file paths can be maximum 260 characters, although Cygwin supports (at least) up to 1024. But more important is the file locking: (modified) SQLite on Cygwin uses the Windows locking mechanism, because it has to cooperate with Windows applications accessing the same database.

Here is an example (full story with explanation) what happens when SQLite uses UNIX locking on cygwin: https://stackoverflow.com/questions/11007024/cygwin-svn-e200030-sqlite-disk-i-o-error

Therefore, on Cygwin, don't use de unmodified amalgamation. Ever ... (please!)

Any chance those Cygwin-specific changes ever land upstream?

Jan Nijtmans (jan.nijtmans) wrote on 2021-03-17 09:36:35:

On Cygwin, using the Fossil's internal SQLite is a no-go.

Therefore, on Cygwin, don't use de unmodified amalgamation. Ever ... (please!)

You say Cygwin must always use the external SQLite. We can accommodate that in auto.def, which detects Cygwin with [get-define host] . If Cygwin is detected then we can force --disable-internal-sqlite . Would that work?

Do you know if MinGW and MSYS need the same fix?

Dan Shearer

Jan Nijtmans (jan.nijtmans) wrote on 2021-03-17 09:36:35:

On Cygwin, using the Fossil's internal SQLite is a no-go.

The the windows bit of build.wiki seems out of date to me. Jan, you raised the MinGW version bug referred to in the wiki page, but that is ancient history isn't it?

Dan Shearer