I need to setup a stable way to do bidirectional sync with a git repo (not on github but anyway). I tried fossil help and searched the online docs but am not quite there yet.

fossil help git is very ... concise (especially regarding import: "TBD" ;)) and otherwise I found

https://fossil-scm.org/home/doc/trunk/www/inout.wiki

and

https://fossil-scm.org/home/doc/trunk/www/mirrortogithub.md

it seems (I am guessing) the former is sort of obsolete regarding bidirectional sync and one now should use fossil git import/export instead?

the latter page explains how to do the export part and it seems I understand that part (not tried, yet). but the page is silent on fossil git import so the whole information regarding that command seems to be "TBD" :(.

I would appreciate any hint where to find the missing info or an explanation how I can achieve a stable bidirectional sync between my fossil clone of the central git repo and that repo in the future (need to push my checkins to the git repo, need to pull changes by others from the git repo).

thx, joerg

but the page is silent on fossil git import so the whole information regarding that command seems to be "TBD" :(.

That's right, the fossil git command only supports export for now.

I've been using fossil import --git and fossil export --git to sync a repository bi-directionally (here's the GitHub counterpart). I have to take care to back up both the Fossil repository clone and the Git repository (and the corresponding mark files, of course). If I re-clone the Fossil repository, the internal rid values may end up different, and I'll have to figure out a way to renumber my commits (or regenerate one of the repositories from the other, losing commit hashes!).

Here's the shell script I've been using:

#!/bin/sh -e

if [ $# -ne 2 ]; then
    echo "Usage: $0 path/to/repo.fossil path/to/git/checkout"
    exit 1
fi

fmarks="$2/.fossil.marks"
gmarks="$2/.git.marks"

if [ -d "$2/.git" ]; then # target repo already exists
    set -x
    fossil export --git \
        --import-marks "$fmarks" \
        --export-marks "$fmarks" \
        "$1" \
    | git --git-dir="$2/.git" fast-import \
        --import-marks="$gmarks" --export-marks="$gmarks"

    git --git-dir="$2/.git" fast-export --all \
        --import-marks="$gmarks" --export-marks="$gmarks" \
    | fossil import --git --incremental \
        --import-marks "$fmarks" --export-marks "$fmarks" \
        "$1"
else # need to create Git repo first
    set -x
    mkdir -p "$2"
    git init "$2"
    fossil export --git --export-marks "$fmarks" "$1" \
        | git --git-dir="$2/.git" fast-import --export-marks="$gmarks"
fi

It's probably safer with a bare Git repository and a separate checkout; otherwise the local Git checkout needs to be manually reset to the new state of the Fossil repo after importing new commits from Fossil. So far, I've had to edit my marks files only once, not exactly sure why. Thankfully, they are still small.

ah I see, thanks for clarifying the situation. I was not aware of fossil git import not being functional so far. also thanks for sharing your script. will have to look into fossil export/import, then it seems.

If I re-clone the Fossil repository, the internal rid values may end up different, and I'll have to figure out a way to renumber my commits (or regenerate one of the repositories from the other, losing commit hashes!).

For the export case, and if you're feeling brave, you might try this patch to src/export.c. It alters export so that UUID instead of RID is used.

There's no guarantees, but it works for me. So far....

There's no guarantees, but it works for me.

If you can confirm that you've been using that patch successfully, i'll get it merged.

If you can confirm that you've been using that patch successfully, i'll get it merged.

Yes but with these conditions:

• I haven't needed it much lately because i haven't been coding much in the last year
• my repository sizes are quite small.

I've spent a bit of time now relearning how the patch worked since it has been a while.

It changes how the old fossil marks file is parsed, and affects both import and export. (Above i wrote export only because that's the case i used.)

So rather than requiring blob.uuid equals a specific rid (the rid as written in the fossil marks file) it matches any rid in the repository clone. Relaxing this constraint seems safe to me but i'm not an expert here.

Anyway, it's good that others have tried this. I see there's a potential problem in a sibling post which i'll respond to shortly.

Thanks also Stephan for keeping this is mind, i've noticed you've bumped the original post occasionally to give it a bit of attention.

Thank you for the patch! Surprisingly, with the patched Fossil, I'm getting:

$ ./fossil export --git --import-marks .../nodepool.2.git/.fossil.marks .../nodepool.fossil
Non-existent SHA-1/SHA-3 in marks file: c855a092534df4a5c0ad08420666ea7943563314a276606ae337ec35c79466be
error importing marks from file: .../nodepool.2.git/.fossil.marks

...despite the artifact does exist in the repository; it's a file:

$ ./fossil info  --repository .../nodepool.fossil \
>  c855a092534df4a5c0ad08420666ea7943563314a276606ae337ec35c79466be
hash:         c855a092534df4a5c0ad08420666ea7943563314
$ ./fossil artifact  --repository .../nodepool.fossil \
>  c855a092534df4a5c0ad08420666ea7943563314a276606ae337ec35c79466be \
>  | head -n 1
\name{nodepool-package}

I'll try to debug it later.

Non-existent SHA-1/SHA-3 in marks file: c855a092534df4a5c0ad08420666ea7943563314a276606ae337ec35c79466be

This message comes when no blob.uuid matches c855a.... or if there's something up with its rid value.

Since you've shown the artifact exists, what rid values does it have in the marks file and fossil repository?

ie, what's the output of these two commands:

$ grep c855a092534df4a5c0ad08420666ea7943563314a276606ae337ec35c79466be .../nodepool.2.git/.fossil.marks

$ fossil sql -R .../nodepool.fossil "SELECT rid FROM blob WHERE uuid = 'c855a092534df4a5c0ad08420666ea7943563314a276606ae337ec35c79466be'"

If the patch is successfully applied, then any non-zero rid should succeed. A non-modified fossil requires the marks file rid to equal the repository value.

Thanks, yes that's my mistake using uuid before init. I must've only used a binary with the original version of the patch.

Sorry about that.

I've successfully exported recent changes from a Fossil repository to a Git repository using fossil export --git with the patch in the previous post. I've also destroyed the rid numbers in the Fossil marks file and verified that unpatched Fossil then fails to export the repository, while patched Fossil succeeds (and restores the rid numbers in the process).

I've successfully exported recent changes from a Fossil repository to a Git repository using ...

Just FYI: this thread is flagged in my inbox to decrease the odds that it will get lost, but i'm not personally well-versed enough with git (on deliberate purpose) to feel comfortable checking that change in. i've asked in our developer /chat, and will ask again here for those who don't following /chat:

Have we got a git import/export user who would be willing to validate the patch at /forumpost/c0ef0256205ac6f0?

If so - please do!

me again: I have tested this script (thanks again!) and everything seems to work if I start with a nonexistent or empty git repo. my remaining problem (I really really do not know anything about git...) now is:

the central, server-side git repo (again: not on github, but same problem...) is already in existence and not empty. so my fossil repo is a derivative of this existing git repo (created via git clone; git fast export, fossil import).

I now want (I think...) to sync my future work in the fossil checkout/repo against my local git clone (question 1: w/ or w/o a git checkout? is this relevant?) and then +git push+ it to the server. in order to do that it seems that I need to pre-create manually the marks files once I guess to let fossil know from where to start the syncing?

if so, how can I do this?

and last not least: +fossil help export+ says it is deprecated and one should use +fossil git export+ instead -- but it would seem the latter does something different (maintaining a (one-directional) mapping into a git repo and thus cannot be used for a bidirectional sync?

so my fossil repo is a derivative of this existing git repo (created via git clone; git fast export, fossil import)

Did you keep the marks files from that import? If not, you might have to reimport the repository anew, making sure to --export-marks on both the Git side and the Fossil side creating the repository from an export. If you do have the marks files, make sure to use --import-marks and --export-marks on both sides of the operation so that only the new objects are transferred between repositories.

and last not least: +fossil help export+ says it is deprecated and one should use +fossil git export+ instead

That's true, the help text is currently gone. fossil git export was written in 2019, possibly to make the official Git mirror of SQLite (a couple of threads from that era: 1 2), but there wasn't enough need for the mirror to be bi-directional. I think I remember someone saying it was easier than modifying export --git to have all the necessary features, but I can't confirm it now. At least the documentation pages are still there.