I'm sorry Fossil didn't work out for you. It continues to work very well for the project I manage (and for which Fossil was originally written).

If you are encounter problems, perhaps you could share reproducible test cases with us so that we could fix them?

Sorry for the rocky journey you're going through.

Were your problems related to the individual size of the binary objects that you managed with Fossil or to some other aspects?

There're limits to any good tool. Looks like your team tried to push Fossil to some limits. Fossil's user base is of course uncomparable in its size to Git's. Some user problems are harder to generalize on limited occurences.

That's why your experience is still valuable to this project. Hope in retrospect you'd find a chance to restate here the issues you encountered.

causing it to crash randomly.

How are you using that word, exactly? I can make several guesses:

1. The same operation done twice fails first, then succeeds the second time.

2. An operation that fails may succeed when attempted later, with no apparent difference between the two cases.

3. The operation fails reliably until some tiny change is made to the input, with no obvious connection between the input change and whether it fails.

Answering that is only the first step towards a repeatable test case, without which we're reduced to educated guesses about what needs fixing.

If the problem is reproducible on your end, but without any obvious way of recreating the problem short of shipping us your massive repository, how about you run Fossil under a debugger and produce a backtrace? If the problem disappears or changes in nature when run under a debugger, tell us that instead.

It amazes me how often software developers will fail to give other software developers the sort of bug report they'd like to receive themselves. Would you accept "crashes randomly" without a core dump, backtrace, or debug result as a useful bug report from a coworker?

At the very least, can we have the output from the /stat page?

the fossil-to-git conversion I just completed an hour or two ago.

Regardless of how this turns out, I'd like to know if that actually solves the problem. I have no reason to believe Git likes large binary files any more than Fossil does.

Just to point out to anyone who doesn't know this already: "fixing" the limitation of "excessively large" files is not trivial (and, as Warren points out, is far removed from fossil's intended use cases). For many different operations, fossil loads the whole contents of a file into memory. For generating deltas, it can hold up to 3 copies in memory: the original copy, the new copy, and the delta. Because fossil stores most versions of a file as deltas, then it traverses older versions it has to unpack those, which means applying deltas, which means memory costs directly related to the sizes of the versions of the blobs. The delta generator also expects the whole body to be in memory as the delta algorithm scans it, and i cannot comment on how much effort it would be to port it to something which only keeps some portion of each blob in memory.

i.e. this is not something which can simply "be patched". It would require significant reworking in some of the deepest parts of fossil. Even so, i'm not certain if sqlite supports the i/o operations needed for fossil supporting blobs larger than the system memory.

Just curious, have you tried using the unversioned feature? I don't necessarily know if this is the answer, or if multiple copies are still loaded into memory, as pointed out above.

unversioned files

First I tried adding a 3G movie

That'll never work short of a SQLite format change.

The only thing Fossil should be doing here is getting the file size up front and refusing to even try if the size exceeds the limit compiled into that instance of Fossil. Arguably it should refuse for anything beyond the default, for broader compatibility.

If I make a new setting which is the maximum size of a managed file, what should the default value for that setting be? 25MB?

The upper bound for the setting will probably be 1GB.

The limit is essentially the amount of memory which fossil is allowed (by the system) to allocate, ignoring (for the moment) any 32-bit overflows which are likely a problem (those are solvable at the code level). Fossil cannot reliably calculate, in advance, whether a given file's size is a problem. It can only try and see what happens. Even specifying some hard limit (say, 2GB) would fail for small devices like Raspberry Pis with 512MB memory and 1GB or less of swap space. (Yes, some people use fossil on such systems.)

Maybe it would make sense to revisit Fossil error reporting so it would properly describe the failure in such case?

If it's out-of-memory condition, then fail with the message. Looks like presently Fossil silently fails to add the super-large file.

Most likely this means fixing those 32-bit overflows along the way.

I currently store several large files as unversioned files for the Linux distribution I maintain as a Fossil repository. The unversioned files are greater than 1GB in size in many cases (installer image ISOs).

First I tried adding a 3G movie

That'll never work short of a SQLite format change.

There may be a simple workaround for that. drh, what if the manifest parser is changed to allow more than one F card to refer to a given file name, with the meaning that each reference after the first just adds more content to the virtual file?

On checkout, Fossil could write a file out and close it when it finds the first F card for that file name, just as today, then on each subsequent F card for that file name, open the file in append mode instead. It could keep a hash table mapping file names to Boolean flags to determine whether to open with the append flag or not.

On diff, Fossil would have to keep the diff buffers around when it reaches the end of one file blob, completing the diff only after the last card in the manifest is processed.

Doing this would remove any motivation for a maximum file size setting, since you could stay well away from the SQLite BLOB limit. ZFS, for example, does just fine with a 1 meg maximum block size, with everything larger than that divided up as necessary. The limit only has to be large enough to allow for efficient file I/O, keeping DB fragmentation and manifest size down while not requiring massive I/O buffers.

This design might even let you get around the current 3× file size magnification problem, which in turn might allow repositories with huge files to work on operating systems for 32-bit CPUs which have filesystems that allow 64-bit file offsets. (e.g. ext3 on x86.)

EDIT: This idea would complicate the deltififcation algorithm, but I think the worst that would happen is that there would be hard barriers at the buffer boundaries beyond which deltas can't be computed. If only one byte in a 10 GiB file changes and the maximum blob size is set to 0.1 GiB, you'd end up with 99 zero-size blobs and one small blob to encode the 1-byte difference. (Plus 100 F cards referring to the pieces of the 10 GiB file.) This is still pretty efficient.

If one byte is inserted in that scenario, then I think you'd end up with 100 small delta blobs, with most of them encoding "added one byte at the beginning, removed one byte at the end" due to the shift-by-one nature of the diff. That means deltification only breaks down when you're inserting a large fraction of the maximum blob size, in which case you're probably going to end up with poor deltification even if you did the delta over the entire file in a single pass.

For backwards compatibility, the code shouldn't assume that incoming blobs are of any particular size. If the default maximum blob size is 1 MiB, all four repositories in my maximum blob size table above would have over-size blobs. The new code will have to be written to cope with this: I don't believe you could fix it with a rebuild, since that would change the hashes of any blobs it had to split.

Also, "versioned with no deltas". For example, Microsoft Word and Excel documents, which change constantly, but binary "diffs" are not appropriate.

I don't see the point. You compute the delta, and decide whether you keep the delta or not, based on the size of the delta and some heuristics and thresholds. It shouldn't have anything to do with the type of the file. Office files are ZIPs with a bunch of files inside, and some edits touch only a small subsets of the files inside. It's possible the ZIP changes only in some places. Thus deltas may very well be appropriate. --DD

Unrelated to Fossil, I successfully stored very large files (many gigabytes in size) in an SQLite database by dividing large files into chunks. See this post for details. My point is that it's not necessary to change the SQLite format. Though maybe when you say "SQLite format change" you mean change Fossil's SQL schema.

Sure you can, but why would you though?

Given [1] and the fact the filesystem is faster for larger sizes starting around 1MB?

And the numbers from [1] are for a single-blob "file". Your manually chunked large files would fair worse I'm betting.

You are basically emulating a filesystem in SQLite, which I'm sure is useful and cleverly done, but the code to access the files can't be normal file-based code, but must go through a different schema-aware and SQLite-based layer. Put a FUSE filesystem on top of your layer, and compare the performance of your SQLite filesystem versus the native filesystem using normal system tools like cp, tar, etc... and see what factor you get.

I'm a huge fan of SQLite, yet I still think its inline blobs is one of its weak spots. And I've been saying this for a while on the SQLite list. Especially when one wants to update a blob, rewriting only a small portion for example. But blob rewriting is obviously not a use case Fossil cares much about :)

[1] https://www.sqlite.org/intern-v-extern-blob.html

Chunking wasn't part of my original task. I discovered the file size limitation, along with the requirement for large files, as I went.

My original task was to archive many named/dated versions of a directory hierarchy in a single file. You'd think Fossil itself would be a good fit (ignoring the file size thing, which I didn't realize at the start), but there was the usual pushback against third-party code whilst simultaneously assuming the goodness of in-house code, yet also unquestioning acceptance of code that shipped with the OS, which included older versions of Tcl and SQLite.

Thus, I implemented a mini-Fossil using Tcl and SQLite, then I had to add chunking when I found it to be necessary.

Simply using the filesystem normally (keeping every directory being "archived") wasn't working well for us because of the massive duplication of identical files between directories.

My first version used zip files rather than SQLite, until I found that given the size of my zip files, the performance of making updates to the archive was utterly execrable. It was far faster to just make new zip files every time. Plus, I was losing many gigabytes in duplication between archives.

Having a specialized filesystem tree associating hashes with contents (e.g. the output of fossil deconstruct) wouldn't work either because in the name of Security, they misconfigured their OS so that every file open, read, write, or attribute change was many orders of magnitude slower than you'd expect. Common tasks like untarring a directory (or even recursively changing its ownership!) went from taking seconds to taking hours. And no one is interested in fixing or even looking into why this is, since if the system is inconvenient, that's proof that it's Secure. I'm guessing it was trying to log every filesystem action, and the logger was timing out at every operation. Bonkers.

Thus, it made a lot of sense to put the filesystem inside an SQLite database. This solved all my issues nicely, and performance (both timing and disk utilization) was a rocketship compared to the baseline, but then again, anything would beat out what was being done before.

FUSE wasn't a requirement since I was working with entire archives at a time, never needing to extract just one file.

Hey everyone! This is the OP. It's good to see the conversation this has sparked. I did want to drop in and say that Git is indeed working for me where fossil did not. I know one of the posters said they had no proof that Git would work any better in this usecase than fossil. It is running significantly faster, however, and so far without error. Commits could take 2-4 minutes, sometimes longer, on my local machine, not counting autosync. So far Git has never taken an entire minute. Of course this is, if I understand correctly, because fossil is safer? Anyway, I just wanted to give that quick update.

Happy to hear about your experience. Keep in mind that you can disable the checksum feature in Fossil, which would allow your commits to be created much quicker.

Anyway, now that git is working well for you, how many times have you had to revert to a previous large binary file?

Good to know that you've settled on a tool that works well for your purpose!

Now that you're back on track, could you list the the details of the Fossil issues that you expeirenced?

I understand from your posts that your Fossil repo size was rather large ~4GB, and you experienced long commit times ~4min. Also there were some 'catastrophic' occurences to your large repos.

• Were you versioning files that were large individually? How large?
• Would a commit take as long on any change (even to a single non-binary source)? Or the extra long waits were only when large binary files were being part of the committed changes?
• Were you experiencing long waits for the 'fossil status' to complete as well, or it was not perceivable slower?
• What sort of 'catastrophies' you had (sorry, for the memories)? Was it repo corruption caused by some failed operation (add, commit, merge)? If that was the case was a large binary file involved in the failed operation or the corruption was triggered by some simple non-binary file change?
• Did the gerenral experience/instability begin from the moment of initially populating of the repo? Or it has degraded as your project work progressed?
• Any other observation that may be specific to your project's use of Fossil

This sort of info would help both anyone with a similar intended use-case as yours and also for the Fossil devs to better see how Fossil dealt with it and possibly improve on that.

Thanks!

"[...] Git is running significantly faster [...]"

Not sure this is one the reasons for the performance differences, but DRH often touts for good reason the "pile of files" vs "database" nature of Git vs Fossil, and by DRH's own testing [1], beyond a certain size, SQLite just does not compete with the filesystem.

Also Git was recently made faster for very large repos, thanks to Microsoft's efforts [2].

Perhaps the doc should clearly state Fossil does not scale beyond certain sizes. At work with have a Perforce repo in the TBs, those do happen in the corporate world.

FWIW. --DD

[1] https://www.sqlite.org/intern-v-extern-blob.html
[2] https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/