I've been running and testing the  code from the clone-resume branch for
a couple  of weeks  now trying  to clone the  massive NetBSD  src fossil
repository[1]. I didn't think it would take as long as it did or I would
have probably paid more attention to  when I actually started the clone,
however, I think I can get  the picture from the /rcvfromlist page which
has 41 entries, the first of which is:

1 	2023-11-29 04:57:18 	amb 	sha3 	163.172.4.16

Given that initial entry, I've been  trying to clone this repository for
over  2  weeks.  I  believe  that  the  41  entries  each  represent  an
interrupted  clone and  resuming thereafter.  Most of  the interruptions
were self-inflicted  as I tested  the ability to resume,  however, there
was one unexpected 30 minute network outage that interrupted the clone.

The Rebuild  phase of the clone  started sometime on Monday  the 11th of
December and  just finished  sometime today, the  16th of  December. For
some reason,  the output stopped  reporting on  progress when it  got to
48.9% and just sat there for  3--4 days. It may have continued reporting
sometime during the night when I wasn't paying attention.

The delta  compression phase took  much less time  than the rest  of the
rebuild but still took all day today to complete.

Here is the output from the last resumed clone operation followed by the
rebuild  and  delta  compression.  The  reason why  there  are  2  lines
representing the percentage complete is  because I suspended the rebuild
so I could do something else with the computer because it was taxing the
I/O quite a bit.

Clone done, wire bytes sent: 28323  received: 1214181826  remote: src.fossil.net
bsd.org (163.172.4.16)                                                          
Uncompressed payload sent: 10834  received: 1214152124                      
Rebuilding repository meta-data...
  48.9% complete...
  100.0% complete...
Extra delta compression... 757855 deltas save 11,729,081,754 bytes
Vacuuming the database... 
project-id: 3109ce34a3d43fd786dcf0133c6a96a6de40c573
server-id:  fbbbc5a67d6d465dcef5d86f1d05fe52d2352a1e
admin-user: amb (password is "BMhnetHYKr")

Here are some statistics:

$ time fossil dbstat -R netbsd-src.fossil 
project-name:      NetBSD src
repository-size:   50,270,945,280 bytes
artifact-count:    4,304,686 (stored as 421,870 full text and 3,882,816 deltas)
artifact-sizes:    60,499 average, 29,093,223 max, 260,430,148,484 total
compression-ratio: 5:1
check-ins:         1,954,758
files:             526,433 across all branches
wiki-pages:        0 (0 changes)
tickets:           1 (1 changes)
events:            0
tag-changes:       0
latest-change:     2023-12-11 21:57:41 - about 5 days ago
project-age:       11,479 days or approximately 31.43 years.
project-id:        3109ce34a3d43fd786dcf0133c6a96a6de40c573
schema-version:    2015-01-24
fossil-version:    2023-12-08 15:30:14 [29e9e84a1e] [2.24] (clang-13.0.0 )
sqlite-version:    2023-11-24 11:41:44 [ebead0e723] (3.44.2)
database-stats:    6,136,590 pages, 8192 bytes/pg, 0 free pages, UTF-8, delete mode
    0m10.89s real     0m04.55s user     0m05.76s system

Here is my attempt to pull today after rebuild was complete:

$ fossil pull -R netbsd-src.fossil 
Pull from https://src.fossil.netbsd.org/
Round-trips: 4   Artifacts sent: 0  received: 116
Pull done, wire bytes sent: 6366  received: 49913  remote: 163.172.4.16

Andy

[1] https://www.fossil-scm.org/forum/forumpost/8cbc83e5d08a86b4

> but times afterall were around 40 seconds - meaning it's somewhat usable

It's definitely useable  if one has the patience to  clone it. I haven't
pulled from  the repository since December  18, so I decided  to "fossil
pull" today to get more of the repository. It took 8 minutes to complete
the pull (which is largely a function of network latency and bandwidth I
imagine):

$ fossil pull -R netbsd-src.fossil                                            
Pull from https://src.fossil.netbsd.org/
Round-trips: 15   Artifacts sent: 0  received: 5718
Pull done, wire bytes sent: 279596  received: 24921431  remote: 163.172.4.16

And then it took another minute or two before it output that there was a
fork:

***** WARNING: a fork has occurred *****
use "fossil leaves -multiple" for more details.

Navigating the timeline is fine as far as I can tell.

I tried  a big diff  from the  UI (clicking two  nodes) and I  must have
picked some pretty nasty ones:


Difference From 738b8b5e0c633827 To 3f7ad32ff303270b

It took  Fossil a  couple of  minutes to generate  a response  and about
1.5GB of RAM.

Firefox, on the other hand, took 9GB of RAM to render the page, but then
suddenly Firefox  stopped outputing anything...  I saw this  output from
Fossil:

------------- 2024-02-17 21:23:54 UTC ------------
panic: Timeout after 600 seconds during web-page reply - user 208,420,000 µs, sys 26,610,000 µs
HTTP_HOST=localhost:8080
HTTP_REFERER=http://localhost:8080/timeline
HTTP_USER_AGENT=Mozilla/5.0 (X11; OpenBSD amd64; rv:121.0) Gecko/20100101 Firefox/121.0
PATH_INFO=/vdiff
QUERY_STRING=from=738b8b5e0c633827&to=3f7ad32ff303270b
REMOTE_ADDR=127.0.0.1
REQUEST_METHOD=GET
REQUEST_URI=/vdiff?from=738b8b5e0c633827&to=3f7ad32ff303270b

So it looks like Fossil gave up waiting for Firefox to consume the data.
I wonder if I  can tune the timeout so that I can  actually see the page
complete.

But it seems that Fossil did just fine otherwise.

Andy