I would like to migrate some of my repositories from Git to Fossil. However, it seems that git-submodules or Git-LFS aren't supported, right? (I could workaround LFS, but I use submodules quite a lot.)

Also: The suggested git fast-export --all | fossil import --git new-repo.fossil doesn't work with Powershell on Windows. It does however work fine with the Windows Git-Bash for example. That wasn't really obvious to me and I think it's helpful to be clear about this in the docs.

I think difficulties and the need to use Git for Windows are more or less generally implied in the relevant part of the "Fossil vs. Git" document.

I suppose you mean this part? I don't see how that explains the PowerShell problem.

The only problem I'm aware of in PowerShell is the lack of a "redirect in from" operator, <. Piping between a stdio type source and sink should work fine.

Me not being a Windows user except under duress leaves me with a nearly uninformed guess that is that this is due to someone playing games with temporary files under an obsolete perception that "Windows doesn't have pipes." Fossil itself does this in certain cases.

If there's a good reason why this should not work and cannot be expected to work in the future, I'll be happy to explain the problem and the workarounds in the docs.

Colin Percival's bsdiff can perhaps be useful to add binary file support in fossil with minimal delta growth. For projects that involve image assets and such, tracking versions is still useful. Of course, very large files would still a problem and though bsdiff might still be useful for those (not sure about memory usage), the sqlite blob limits would kick in.

Fossil already has a binary diff algorithm. Switching to bsdiff wouldn't solve the problem described in the linked document.

This problem has already come up before, see:

• https://fossil-scm.org/forum/forumpost/04acf74ef7
• https://fossil-scm.org/forum/forumpost/162cd28efc

Ouch!

This being an inherent design issue with PowerShell that Fossil cannot fix, I've expanded the relevant doc, adding the new "Converting Repositories on Windows" section.

Does this work, or are improvements needed?

For me, it works!

I was about to mention that since PowerShell is available for other platforms, this could affect other systems as well -- but it looks like at least PowerShell on Ubuntu doesn't have this problem.

While that does mean the "on Windows" qualifier on my section name covers us here, doesn't that undermine my theory of the core problem? It cannot be that it inherently pulls everything into RAM, decides how to process it, then sends it out if the Ubuntu port doesn't exhibit the same symptom.

While slagging on Windows has a certain amusement value, I prefer to be correct when I indulge in this pleasure. 😛

Yes, this seems somewhat strange, indeed.

But the only case where output is delayed until all input is read is PowerShell on Windows piping to an external program.

This seems not the case on Ubuntu, where the external program starts processing immediately (verified this by piping though sh -c cat -n to make sure more and similar are not aliases to internal PowerShell functionality).

I wish I knew the why of this, but regardless, I've decided to dial back the strength of the new prose, to speak only of facts actually in evidence.

This would some time soon also work on Windows. Of course, you will need to install the latest PS 7 then...

So…it's not related to data volume at all, but to data encoding? It's possible to make the conversion choke even with an all-but-empty repository?

So…it's not related to data volume at all, but to data encoding?

Yes, to my very surprise! Following two tests with https://github.com/drhsqlite/fossil-mirror.git and PowerShell 5.1.22000.282 on Windows 11.

A: Export all from Git → Fossil: full pipe buffering causes graceful OOM.

PS C:\...> git fast-export --all | fossil import --git test1.fossil
fatal: Out of memory, malloc failed (tried to allocate 4203555 bytes)
Exception of type 'System.OutOfMemoryException' was thrown.

B: Export only part from Git → Fossil: data encoding problem.

PS C:\...> git fast-export master~1..master | fossil import --git test2.fossil
]ad fast-import line: [blob

So my earlier assumption that PowerShell's full pipe buffering may cause the limit proved wrong. The strange error message "]ad fast-import line: [blob", quite similar to the earlier example "]ad fast-import line: [JSON", tricked me.

I dunno; it looks like there are two failure cases here.

Thanks for testing and reporting. I would have never bothered to fire up a Windows VM and try it myself, but I am glad to know the answer.

I dunno; it looks like there are two failure cases here.

Yes, there are, but so far I think only case B was reported on this forum.

Anyway, as soon as PowerShell will connect pipelined processes directly, either problem should be solved.

Now I see where the strange error message comes from: PowerShell injects CR before LF, which get trimmed to CR alone, and instruct the terminal to move the cursor back to the start of the current line.

It's interesting to note that PS issue #559 about this problem was created in 2016, and the fix was implemented in the beginning of 2017, so it looks like PowerShells of versions ≥ 6.0 should work more sensibly.

... I use submodules quite a lot

Not to discourage your choice of migrating from Git, however I wonder if you considered just transitioning to Fossil instead of full-history migration.

Basically, select a list of recent releases and just port them by checking out from Git and re-commit them into Fossil repo using some custom scripts. It may be possible to maintain some sort of xref to Git commit-id via Fossil tags.

This approach should also take care of submodules naturally per version integrated into the main repo release.

In case more granularity needed, the Git repo(s) are still there for continuity.

Just an idea.