it's not something urgent, but i'd like to know why it is so.

Though i cannot answer your question, this might be a workaround until it's resolved:

Reposurgeon can convert from svn to git, or even possibly between svn and fossil (probably using git as the intermediary), but its "recent changes" list says:

• Fossil import/export from checkouts is supported; --format=fossil is gone.

Whether that means that it can now only import/export fossil via a checkout, and what exactly the implications of that are, is unclear to me.

PS: that's not a product recommendation, per se, as i've never used it.

thank you! simple importing SVN repo via git svn clone works too. i've never used Reposurgeon too, looks like it could do it without full conversion.

but i am interested in what is so different between two kinds of importing, and why Fossil constantly eats memory with SVN import, but not with git import. something strange is going on there.

my wild guess (not checked, just guessing) is that git exports each commit as a set of new files (this fact is not checked), and SVN exports as deltas (this fact is checked, SVN does exactly that when dumping a repo) to some previous commit. so Fossil eats up more and more memory while reconstructing files from SVN dumps as deltas goes further, and eventually runs out of memory.

but shouldn't content cache take care of that? as we only need to reconstruct one file at a time, and we don't need to keep the old file version after applying a delta, it shouldn't be a problem... unless all file versions are kept until the very final delta applied. this may be the case, but i'm not yet familiar with that part of Fossil code.

btw. is there any way to log memory allocations in Fossil? i mean, some define or something as easy as this. ;-) as far as i remember, Fossil don't bother to free all resources, but it definitely does free some temp ones. and it would be great to see what kind of resource "accumulates" more and more in any case.

i tried to force content cache cleanup after processing each new SVN revision, but that didn't help. not that i expected it to help, just wanted to rule out "inter-revision cache issues".

now i need to dive much deeper into Fossil code, and it would be great to have some tool to log all allocations. if not by type, then at least by source file position. ;-)

and it would be great to have some tool to log all allocations. if not by type, then at least by source file position.

We don't currently have such a thing but a patch which replaces fossil_malloc() and friends with a macro which passes on __FILE__ and __LINE__ to their real impls, such that we could add that info to the oom crash, would be thoughtfully considered.

p.s.: could you please edit the topic title to "OOMs" instead of "segfaults" (or something)? because this is not technically a segfault bug, it just manifests itself this way.

ok, it seems that i found it! ;-)

looks like the problem is not that hard: svn import never frees delta and target blobs, and they simply accumulate. you only need two small changes to fix it.

find two svn_apply_svndiff() invocations in import.c, and insert:

blob_reset(&deltaSrc);
blob_reset(&target);

right after the call to svn_handle_symlinks(), and before the closing }.

this seems to fix memory leaks, and 600 revisions are imported without unbound memory consumption. i will try to import the whole Sauerbraten repo now (6000+ revisions), and report the results. but i believe that this is it.

call to svn_handle_symlinks() inserts target content into the database and we don't need to have it in memory anymore. and we don't need deltaSrc after callig svn_apply_svndiff(). it is easier to free them both after calling svn_handle_symlinks(), though. and just in case: do not free rec.content there, it is properly freed at the end of the loop.

i am not providing the patch, so you don't have to invent some way to rewrite two trivial calls. ;-) but just in case, i donate my analysis to public domain too (if it even matters from the legal standpoint ;-), you may use it as you like, without any credits.

you only need two small changes to fix it.

Thank you very much for the debugging and report. We'll get this fixed Real Soon Now (very possibly today).

i am not providing the patch, so you don't have to invent some way to rewrite two trivial calls. ;-)

No worries, we excel at reformulating patches ;).

find two svn_apply_svndiff() invocations in import.c, and insert:

Just to confirm, is this what you meant?

Index: src/import.c
==================================================================
--- src/import.c
+++ src/import.c
@@ -1584,12 +1584,16 @@
             }else{
               blob_zero(&deltaSrc);
             }
             svn_apply_svndiff(&rec.content, &deltaSrc, &target);
             rid = svn_handle_symlinks(zPerm, &target);
+            blob_reset(&deltaSrc);
+            blob_reset(&target);
           }else if( rec.contentFlag ){
             rid = svn_handle_symlinks(zPerm, &rec.content);
           }else if( zSrcPath ){
             if ( zPerm==0 ){

yes, exactly! and there is another such code snipped around the line 1630.

yes, exactly! and there is another such code snipped around the line 1630.

Thank you for the report and analysis. After looking those sections over closely, the leak is now obvious and both blocks have been patched on the trunk.

thank you! just to finish with this, the promised report: i succesfully imported 6860 revisions from Sauerbraten SVN, with Fossil comfortably sitting in 20-40 MB of RAM. it took slightly more for the final metadata rebuild and deltification, though (sometimes peaked at 100 MB due to huge binary files), but Fossil released memory properly. final repository size is about 6GB (without vacuuming).