Fossil Forum

Is Fossil still bad for lots of binary files?
Login

Is Fossil still bad for lots of binary files?

Is Fossil still bad for lots of binary files?

(1.1) By jest (jestarray) on 2022-05-15 04:08:14 edited from 1.0 [source]

https://www.omiyagames.com/blog/2014/02/15/farewell-fossil-version-control/

(the comment section in the above post go into more detail)

While Unity itself doesn’t generate super-large files (I think the worst I’ve got was a 2MB scene file), there are a lot of images and sound files I deal with that inflates that number. It’s not uncommon for me to make commits of 600 new files, most of them small images around 0.3 MB. I’ve seen Fossil crash multiple times when making these commits, enough so for me to quit

https://fossil-scm.org/forum/forumpost/b022e1be1f

There have been some posts that state bad experiences with fossil when it comes to dealing with large binary files, in particular, game development because you have to store potentially very large files audio, images, etc files.

Do these problems still exist?

(2) By Warren Young (wyoung) on 2022-05-15 06:58:48 in reply to 1.1 [link] [source]

Part of your answer is here.

Even if you use uncompressed assets and compress only for builds as the article recommends, there’s a cost to cloning 100% of all historical versions of every file in your project, and storing them on each developer’s machine.

You may be better served by a centralized version control system, which sends only one version to each client at a given time.

(3) By Stephan Beal (stephan) on 2022-05-15 07:28:54 in reply to 1.1 [link] [source]

I’ve seen Fossil crash multiple times when making these commits, enough so for me to quit

The only cases we're aware of where fossil will outright crash in normal use is when it runs out of memory. In order to (greatly) simplify development, its allocator is configured to crash if an allocation fails. This condition is most prominent when processing very large files because applying deltas to files requires having the original version, the delta, and the new version all in memory at the same time.

Similarly, trying to use the zip/tar commands to export a repository which contains "too much stuff" will cause it to run out of memory because those archives are created in memory, but where the boundary of "too much" is depends very much on the repository's contents.

There have been some posts that state bad experiences with fossil when it comes to dealing with large binary files, in particular, game development because you have to store potentially very large files audio, images, etc files.

Warren's answer is, IMO, the correct one: a centralized SCM, rather than fossil's "every repo contains all the history" is a better choice for that type of project. Though fossil can handle binary files just fine, it's not an ideal tool for managing them within a team.

Just a couple of days ago in /chat, Richard posted a link to an SCM which is supposedly used specifically for Unity's own development, which is presumably ideal for your purposes:

https://www.plasticscm.com/

It's a commercial (subscription-based) product, but they have a free demo tier for up to 3 users and 5GB.

(4.1) By mattwell on 2022-05-15 11:39:38 edited from 4.0 in reply to 1.1 [link] [source]

I have been pleasantly surprised at just how far you can take fossil in managing large binary files. A few things you can do:

  1. Where possible manage large files using unversioned. Chances are you don't care about past versions of some of the assets.

  2. If you have a large number of large files to commit and fossil crashes just commit the files in smaller batches.

  3. Break your project into two or more fossils and add a little scripting to keep things aligned.

A someone who knows fossil quite well I'd push the tool to its limits rather than learn a brand new tool. However if you are just starting you might be better off to ramp on the tool designed to solve the problem you face.

One more thing, you might choose to keep occasional snapshots of your assets properly committed (e.g. put them in a "golden" directory) so that should you accidentally remove the unversioned versions you have a distributed backup. This way the fossil database won't grow excessively large but you still have some history.

Just my $0.02

(5) By Felipe L. (ev685rs97mtzw9a6) on 2022-07-14 01:19:45 in reply to 1.1 [link] [source]

It's a pity that Fossil still can't handle well big binary files natively. And there is not tools like "git-annex" or "git-lfs" equivalent.

(6) By ravbc on 2022-07-14 09:57:59 in reply to 5 [link] [source]

There is not a single DVCS, that I'd know of, that can handle well big binary files natively. Big binary files and DVCS just don't play well together. Nevertheless fossil uv really helps with dealing with large files in a distributed way.

Moreover git-annex and git-lfs are not part of git - they are extensions, which git makes easy to plug into its command line. Port them to fossil, and you'll get the same functionality. Maybe with the exception of the combined command line...

(7) By Stephan Beal (stephan) on 2022-07-14 10:08:26 in reply to 6 [link] [source]

There is not a single DVCS, that I'd know of, that can handle well big binary files natively.

Plastic SCM, mentioned up-thread, is a niche commercial SCM which targets Unity1 developers, and specifically addresses their use case for huge media blobs.

Sidebar: fossil's limit of blob sizes is tied (A) to sqlite3's limits and (B) to RAM, as it frequently pulls whole blobs into memory and requires two copies at once when applying deltas (which happens almost every time a historical version of any file is examined in any way).

Port them to fossil, and you'll get the same functionality.

Patches are always thoughtfully considered :) (but (full disclosure) not always accepted).


  1. ^ a popular game development platform

(8) By skywalk on 2022-07-14 13:21:40 in reply to 5 [link] [source]

As SQLite claims superiority over fopen(), I use thousands of sqlite db's.
(And ATTACH is my friend, though not in the thousands!)
Why can Fossil not do the same?

(9) By Stephan Beal (stephan) on 2022-07-14 15:21:21 in reply to 8 [link] [source]

Why can Fossil not do the same?

Fossil's current behavior is well within its intended usage and its design. It's not designed to be used for multi-gigabyte blobs (binary or otherwise). It's designed to be used for source code and similar text-heavy projects which might have a few blobs to go along with them.

No SCM can be all things to all people, and people who need big-big blobs are one group to which fossil's design simply doesn't cater. Such is the way of software: every feature and design decision implies certain inherent limitations.

That said: patches to make is usable by Unity-based game designers and the like would certainly be thoughtfully considered (noting that adding such support goes very much the current internals, which were not designed with sharding of blobs in mind, so writing such a patch would be an uphill battle from the start).