There are two central problems in making arbitrarily-large Fossil repos practical:

1. The total amount of data. This is the easy problem. If the Internet can support Netflix, it can support large Fossil repo clones.

2. The post-clone rebuild time, in which Fossil re-creates all of the lookup tables that give it capabilities such as rapid forward tracing through the repo history and invulnerability to Git's dread detached head state.

We can easily see that the second problem dominates by comparing the time to scp a large Fossil repo to the time taken to fossil clone it over the same network.

I believe the solution to this problem is shallow clones. Once we can find someone with the time, will, and skill to implement the feature, the primary problem that developer will face is that reconstructing the current version of a given file can require pulling up every historical delta artifact.

That isn't a necessary consequence of Fossil's repo format. This need is simply due to Fossil's current implementation, which prizes overall smallest size in the repo. Thus fossil rebuild --compress.

To make shallow cloning practical, Fossil would need logic that puts a limit on how far back the delta chain for a particular file can go. If you set it at 1 year and change each file only once a year, then Fossil's repo size will end up not having any delta compression at all. The upside of that is that you can clone only the past year's worth of repo data for the cost of only about one checkout's worth of data.

The shorter you make that window, the bigger the size of a full repo clone, since delta compression becomes increasingly less effective.

It would be possible to write this feature in a way that lets you set the normal delta window size limit to 30 days but increase that window size by some nonlinear function like 2ⁿ, so that the prior window is 60 days, doubling back to the project's founding date, where the oldest delta window would be roughly half the repo's total history.

That would reduce the amount of repo bloat needed to support this feature, but it would require occasional rebuilds, else the current 30-day delta window just keeps rolling over indefinitely, creating a string of 30-day delta windows.

The end goal of all of this is that you should be able to tell Fossil to pull only the most recent versions of all open branches, going back no deeper than the current delta window. Fossil will then model the common use case: we mainly need the current versions immediately, maybe a few days old occasionally, and older versions with increasingly smaller probability. You might need all versions eventually, but not immediately after the initial clone. Historical delta windows can be pulled in over time as required.

Further information:

The delta algorithm actually works pretty well. If you take a tarball of the latest Fossil check-in, it is about 5.4 MB in size. But the whole repository with 12,634 check-ins is only 48.3 MB (if you omit the UNVERSIONED table) or less than 9 times larger.

So, from one point of view, if you simply encode the first 9 check-ins as full-text, you get the other 12,625 check-ins (99.92% of all check-ins) for free. :-)

If fossil scrub can get the attention it needs, an interim "big clone" strategy could be implemented.

I envision something like:

At main host:

• clone main repo
• scrub clone
• setup job to periodically update clone from main repo

At clone host:

• make local copy of scrubbed clone
• fossil sync https://example.com/main_repo

This would enable contributors to projects with large repositories to quickly make local clones.