Suggestion: Order-independent fossil sync --all

(1.1) By Thomas Hess (luziferius) on 2024-03-21 15:43:48 edited from 1.0 [source]

Hi forum,

I'd like to suggest an improvement for the sync --all behavior:

The current algorithm for sync --all boils down to basically:

for remote in remotes:
  sync_with(remote)

While it does sync with each remote, results depend on the order the remotes are visited. If a pull from a later remote downloads new artifacts, it takes up to N runs of sync --all to get all remotes to be in sync (where N is the number of registered remotes).

I suggest revisiting the algorithm, and replace it with:

for remote in remotes:
   pull_from(remote)
for remote in remotes:
   push_to(remote)

This algorithm will collect all new artifacts from all remotes first, build the union locally, and then broadcast the collected artifacts to all remotes.

It will ensure all remotes are in sync with one call of sync --all

Greetings, Thomas

(2) By Stephan Beal (stephan) on 2024-03-21 20:45:09 in reply to 1.1 [link] [source]

I'd like to suggest an improvement for the sync --all behavior:

Just FYI: we discussed this in /chat and are in agreement that it would be a nice improvement, but its implementation is hindered by old design decisions which would need to be resolved before your suggestion can be implemented.

(3) By Thomas Hess (luziferius) on 2024-03-21 23:19:04 in reply to 2 [link] [source]

I haven't looked at the internal implementation, so unsure if this below is more feasible.

How about shoehorning the behavior into the existing loop:

queue = Queue()
queue.append_all(remotes)
visited = list()
while not queue.is_empty():
    remote = queue.pop()
    received_artifacts = sync_with(remote)
    if received_artifacts > 0:
        queue.append_all(visited)
        visited.clear()
    visited.append(remote)

This alternative algorithm achieves the same result with more internal work and higher complexity, but is designed to take a list of remotes, and only uses the sync action. (It may require extending whatever function implements the sync to return the number of received artifacts)

I'd personally say this algorithm is way worse than the first proposed, so I'm fine with you rejecting it :)

(4) By Richard Hipp (drh) on 2024-03-21 23:27:48 in reply to 3 [link] [source]

The earlier algorithm that I tried (and failed) to implement looks at the number of new artifacts that are received back from each sync attempt. It only does a second pass for remotes that occurred earlier than the last remote that returned a new artifact.

So for example, if you have four remotes, and you do a "sync --all", but only the second one sends back new content, then on the second pass you only need to repeat the first remote. Even if the last remote sends back new content, you only need to repeat the first three. So you end up doing fewer HTTP requests.

Having just taken as second look at the code, I now think I might see around the legacy data structure problem that thwarted my first attempt to make this happen.

(5) By Thomas Hess (luziferius) on 2024-03-22 22:26:03 in reply to 4 [link] [source]

I saw you implemented a two-pass mode for sync --all. Thank you for implementing my request :)
I'll compile from source and test it.

I have two mirrors for one of my projects, a "private" copy on my home server Raspberry Pi, and a "public" one on chiselapp.com. I mostly use the local mirror, but sometimes there's activity on chiselapp. There is no clear hierarchy between both instances, so I often ran the sync twice to sync them up before coding sessions. This change will help me on my setup.