Hi,

I noticed that Fossil gets extremely disk space hungry and extremely slow when dealing with larger repositories.

In particular, cloning https://pkgsrc.fossil.netbsd.org takes an order of magintude longer than cloning the same via Git or Mercurial. The resulting repo file is also several times bigger than both Git or Mercurial.

Any idea what's up with that? I could understand being a few percent bigger, but several times?

Also, any operation on this repo takes very long - to the point that CVS, that has to talk to the server, is almost faster.

Any idea what's up with that? I could understand being a few percent bigger, but several times?

Though fossil can architecturally handle it such repos, it's far from optimized for those scales. Fossil was designed for small-/mid-sized projects, not behemoths like that one.

One of the culprits is the number of files. Every checkin contains a list of all files which are part of that checkin, regardless of whether they were modified.

That particular repo uses delta manifests, which shrinks their size but not the logic needed to create and traverse them. Its current tip is:

https://pkgsrc.fossil.netbsd.org/artifact/18aed899a9074f9f

Which is a delta manifest derived from:

https://pkgsrc.fossil.netbsd.org/artifact/ba7dfd1855ecbc

which is a ~10.2MB manifest with 103006 files in it. Every commit (or call to "status") has to traverse every one of those files in order to be able to know for certain whether they've been modified or not. Even on an SSD, traversing those will take time.

Another culprit for large repos is the repo-cksum setting, which causes an extra layer of hash defense to be calculated during manifest generation, and that calculation's cost is linear on the number of files in the checkin and their sizes. It's also a massive memory hog. Because of the cost, and because it's the last in line of 2 or 3 other levels of hash protection, that calculation can be disabled via the repo-cksum setting.

There are probably other factors which come into play with massive repos, but those are the two which immediately come to mind.

That's nearly a 10GB repo with 1,861,916 artifacts.

https://pkgsrc.fossil.netbsd.org/stat

This example repo has come up a few times on the mailing list before. You can search to find responses there, but Stephan's is pretty much what I remember others pointing out as well.

See also this forum post to find an answer

Are there any plans to make repos like this work?

I'm mostly asking because I am a pkgsrc contributor, and would like to use Fossil instead of CVS. But cloning took already several hours and the resulting checkout is so gigantic that it wasted too much SSD space to be viable.

FWIW, https://src.fossil.netbsd.org/ has similar problems.

// Edit: Also, Fossil is what the current GitHub mirror is generated from, as well as the Mercurial mirror. Which sometimes takes hours to update.

Are there any plans to make repos like this work?

Very probably not until someone who is directly affected by such extreme cases decides to get involved and can upend the architecture to be performant at such scales.

In all honesty, i can't understand why those particular trees insist on using fossil, as it's never been a great match for their needs.

Easier working with the young alive repository with a much better performance would remain as the win-win from these efforts.

If your repository has 1 million files then splitting it into old won't matter much for performance. The age of the files has no real effect on the processing speed of manifest creation (provided it's always traversing the latest copy - older copies are more expensive to traverse), but processing 1 million files of any age is going to take a tremendous amount of time above and beyond any SCM-side work.

Try the following in your shell, which simply runs "stat" on a single file 1 million times:

$ date; i=0; while [[ i -lt 1000000 ]]; do stat wiki.c > /dev/null; i=$((i + 1)); done; date
Wed 09 Sep 2020 01:49:18 PM CEST
^C

i cancelled it after 7 minutes on this laptop (with an SSD drive - a spinning disk could be considerably slower).

Re-running that test with 1000 files takes 3 seconds and 10k takes 29 seconds. 100k will probably take almost 300 seconds and 1M probably 10 times that. Best case, nearly an hour just to stat that many files.

That's without any of fossil's own internal work going on, and fossil has to do a great deal more than stat files when creating a manifest.

cloning...takes an order of magintude longer than cloning the same via Git or Mercurial.

It isn't an apples-to-apples comparison. Fossil's doing more. While there may be some micro-optimizations available to make it somewhat faster, the only ways to make it give the same performance would be to throw away some of those advantages.

1. Cloning probably isn't the biggest time, in fact, it's rebuilding the metadata required by Fossil's superior data model. I don't know about Mercurial, but Git doesn't need that step because its data model only lets it trace backwards up toward the root, not the other way. This has the bad effects laid out in that document and elsewhere.

   The only way I see to avoid this cost is for the server to somehow transmit its precomputed metadata down to the client, which can then slurp it in uncritically.

2. Fossil has more checksums, cross-checks, and referential integrity measures than Git. Maybe Mecurial, too, though that's just speculation. You've been shown the repo-cksum option, but that's the only one you can turn off. The rest of the costs you don't get a choice on, currently. And if you did, would you turn them off?

3. Later, you bring up CVS, but that's apples-to-kumquats. Fossil could approach similar performance by allowing shallow cloning (item 21), but that's been on the wishlist for years now.