A simple approach would simply be to commit each round-trip and not error
out when  a network error  occurs and then  just allow "fossil  sync" to
do the rest, however...

While  experimenting  with the  "fossil  purge"  command, I  emptied  my
repository entirely  to see how well  "fossil sync" could then  clean up
the mess (as I mentioned here):

https://www.fossil-scm.org/forum/forumpost/c1ff250092

I then ran "fossil sync" and here is what it did:

$ time fossil sync --httptrace
Sync with http://localhost:8080/
Round-trips: 2   Artifacts sent: 0  received: 0
Server says: pull only - not authorized to push
Round-trips: 4725   Artifacts sent: 0  received: 34202
Sync done, sent: 4177319  received: 54810455  ip: 127.0.0.1
***** WARNING: a fork has occurred *****
use "fossil leaves -multiple" for more details.
/***** Subprocess 31921 exit(0) *****/
   34m57.93s real     2m44.71s user     1m31.62s system

So it took 35 minutes to "clone"  via the sync protocol (and tested with
and without --httptrace, and the use of the option didn't introduce much
of significance). It would seem that  the sync protocol ended up sending
one "card" per round-trip for a considerable amount of round-trips.

I also found  that after the big synchronization, that  the clone didn't
have all  of the  artfacts that  existed in  the parent  repository. For
example,  "fossil dbstat"  shows that  in  the parent  there are  13,055
checkins, however,  in the clone  that I  purged and then  synced, there
are  only 11,719  checkins,  and 38,760  artifacts  vs 44,088  artifacts
respectively.

I used "fossil deconstruct" to attempt to  see if there is a pattern for
missing  artifacts. One  such missing  artifact is  000575c77d which  is
named in cluster artifact 5e1353f514 which *is* present in the clone, so
it's odd that sync wouldn't have discovered that.

Another example, again  shows that artifact 0008f29c769  is missing from
my  clone  and  it  is  again  part  of  a  different  cluster  artifact
9b5084bbc2.  Perhaps there  is  a bug  in  processing cluster  artifacts
because my  clone clearly has the  cluster which should have  produced a
phantom in the phantoms table which  eventually resulted in a "gimme" to
pull the content, but there are hardly any:

$ echo "SELECT count(*) FROM phantom;" | fossil sql 
5

I also found that in the clone, the unclustered table is quite large:

$ echo "SELECT count(*) FROM unclustered;" | fossil sql
34207

Consequently, the next  time I tried to  sync, my client sent  a list of
"igot" cards that was almost as large:

$ grep -c  '^igot' http-request-1.txt
34203

Perhaps this is  because there just are a lot  of missing artifacts from
this repository.

Finally, I decided to try --verily:

$ fossil sync --verily --httptrace
Sync with http://localhost:8080/
Round-trips: 2   Artifacts sent: 0  received: 0
Server says: pull only - not authorized to push
Round-trips: 8   Artifacts sent: 0  received: 5328
Sync done, sent: 2257296  received: 13046887  ip: 127.0.0.1
/***** Subprocess 58152 exit(0) *****/

It ran in much less time (I forgot to time it, but it was probably about
60--120 seconds at the  most) and appears to have brought  in all of the
artifacts, though there is still one less checkin which is odd:

artifact-count:    44,088 (stored as 17,033 full text and 27,055 deltas)
check-ins:         13,054

So, given  this... perhaps we could  track a failed clone  in the sqlite
database  in  the Fossil  repository  and  the  next  time that  a  sync
operation causes a pull the  client could send the "pragma send-catalog"
to resume the clone operation? Something like:

https://www.fossil-scm.org/home/info/ec26471439ec5294

Let's see how this works...

$ fossil clone http://localhost:8080/ clone.fossil
Round-trips: 3   Artifacts sent: 0  received: 25474
response truncated: got 1964529 bytes of 5001092
Clone done, sent: 1009  received: 17379269  ip: 127.0.0.1
server returned an error - clone incomplete
Rebuilding repository meta-data...
  100.0% complete...
Extra delta compression... 
Vacuuming the database... 
project-id: CE59BB9F186226D80E49D1FA2DB29F935CCA0333
server-id:  45821d85f56221cd472fd4e08ddc7fc7babccf5f
admin-user: amb (password is "RuXG64Bs9C")
clone operation had errors. Run "fossil pull" to complete clone.

So far so good, right?

$ fossil pull -R clone.fossil
Pull from http://localhost:8080/
Round-trips: 1   Artifacts sent: 0  received: 200
infinite loop in DELTA table
Abort trap (core dumped) 

Looks like this approach  has some bugs. I'm not yet  sure where the bug
is---is it in the rebuilding of  the partially cloned repository? Is the
cache somehow corrupted? Is it in the calculation of this number:

https://www.fossil-scm.org/home/artifact?udc=1&ln=273-275&name=d49bd600d03d5de7

$ echo "SELECT max(rid) FROM blob;" | fossil sql -R clone.fossil
25702

Guess I'll have to dig into this a bit more.

Andy

> So,  given this...  perhaps  we  could track  a  failed  clone in  the
> sqlite database  in the  Fossil repository  and the  next time  that a
> sync  operation  causes a  pull  the  client  could send  the  "pragma
> send-catalog" to resume the clone operation?

Or maybe this would be a better fit as an appendage to the "fossil clone"
command; perhaps a --resume option could be added.

Andy

I agree  that ideally a partial  clone should be treated  no differently
than  one that's  just been  offline, however,  there are  actually some
useful optimizations that clone does which the normal sync protocol does
not  use  except upon  request/demand  of  the  user (because  it  isn't
expected to need tens of thousands of artifacts).

Also, if you read  my analysis of the "need a  lot of updates" scenario,
wherein  I  completely  emptied  my  repository  so  sync  had  to  pull
*everything* again, I appear to have  exposed a bug in the sync protocol
that  will need  to be  figured out.  This is  why I  opted to  make the
"resume" behave more like a clone than a sync.

I  suspect that  the current  failure  to "resume"  in the  clone-resume
branch may  be related to how  Fossil rebuilds and constructs  the delta
table at  the end of  the the  clone---maybe rebuild should  be deferred
until clone is complete?

Thanks  for your  feedback regarding  "auto-resume". Are  you suggesting
that  perhaps one  simply need  to call  "fossil clone"  as second  time
giving the same arguments  and it will pull in the  rest of the content?
Currently, Fossil  complains if you  try to clone  on top of  an already
existing file and I think I  would prefer to preserve that warning which
means  the user  would have  to pass  in another  argument. I  suggested
"--resume" because the intent is clear.

Or did you have something else  in mind for "auto-resume"? Maybe instead
it could just try harder? For  example, with autosync, it is possible to
configure autosync-retries which will cause  the sync to try harder when
synchronizing  content.  Maybe  you  mean  that  clone  should  also  do
something similar  rather than simply  bailing out on the  first network
failure?

Thanks,

Andy

> The one  that I'm  hoping ends up  far more common  is that  the clone
> aborts at  a point where  you can still  "open" the repo,  because you
> have  enough pulled  down that  Fossil  can still  give you  something
> useful.

I've also had a similar thought. For example, if the clone aborts before
the clone is able to receive project-id,  I don't think we have a useful
clone from which a resume can hope to be successful. Consequently, I was
thinking  of  retaining  the  original behavior  if  the  project-id  is
not  defined after  the  clone completes  (either due  to  full sync  or
interrupted sync).

> A manual sync: fossil sync -R /path/to/repo.fossil. You should be able
> to run  that with further partial  syncs until fossil open  is able to
> succeed, per above.

That's how it  currently works in the branch (well  at least is intended
to work), however, there appears to be something wrong and I haven't had
time to investigate.

Thanks,

Andy