Fossil Forum

Forum
Login

Infinite loop in DELTA table

By andygoth on 2018-09-22 00:18:13 [link]

I'm having problems cloning the SQLite repository. I get an infinite loop error, as well as virtually empty repository (despite being 20 megabytes).

$ f version
This is fossil version 2.2 [613fe1b1fb] 2017-04-19 07:08:01 UTC
$ f clone http://server/sqlite/ sqlite.fossil
Round-trips: 3   Artifacts sent: 0  received: 13104
Clone done, sent: 912  received: 19401205  ip: 130.210.192.30
Rebuilding repository meta-data...
  100.0% complete...
Extra delta compression...
Vacuuming the database...
project-id: 2ab58778c2967968b94284e989e43dc11791f548
server-id:  02b60258c7ddda07b2551f01f0673f63fa51b74c
admin-user: user (password is "password")
$ f sync -R sqlite.fossil
Sync with http://server/sqlite/
Round-trips: 1   Artifacts sent: 0  received: 208
Fossil internal error: infinite loop in DELTA table
$ f info -R sqlite.fossil
project-name: SQLite
project-code: 2ab58778c2967968b94284e989e43dc11791f548
check-ins:    1
$ f info -R sqlite.fossil trunk
uuid:         704b122e5308587b60b47a5c2fff40c593d4bf8f 2000-05-29 14:16:00 UTC
tags:         trunk
comment:      initial empty check-in (user: drh)
$ f rebuild -stats sqlite.fossil
  100.0% complete...
Artifacts:       13062
Manifests:           1
Clusters:            0
Tags:                0
Wikis:               0
Tickets:             0
Attachments:         0
Events:              0
Other:           13061
$ f sync -R sqlite.fossil
Sync with http://server/sqlite/
Round-trips: 1   Artifacts sent: 0  received: 208
Fossil internal error: infinite loop in DELTA table

I'm trying to switch to a newer Fossil version, but I'm having trouble with that.

The actual most recent check-in for my SQLite repository I just cloned is d10e63629183f6daf0c263cd4dae052a3786c8c1480b3b6a73124b3315e41951. It displays just fine on the server, which is running Fossil version 7950dc2297.

By drh on 2018-09-22 15:45:53

Can you get me a copy of the fossil repo that is showing the delta loop, so that I can investigate the problem?

Have you tried running "fossil test-integrity" on the repo that shows the problem?

By andygoth on 2018-09-26 01:53:34 [link]

Here is the repository. Please let me know when you've downloaded it because I don't intend to leave it up forever.

test-integrity has heartburn like you couldn't believe. Wrong hash on every artifact, it seems, plus a number of skipped phantoms. At the end, it says:

13062 non-phantom blobs (out of 13316 total) checked:  12001 errors
low-level database integrity-check: ok

I am using Fossil client version 2.2 [613fe1b1fb] to clone SQLite from Fossil server version 2.7 [5fdd53120e]. The SQLite repository's latest trunk version is 42e04fefbc241dd33f12abd66344a87720ae4cda 2018-09-25 13:51:31 UTC.

By andygoth on 2018-09-26 03:49:58 [link]

Amazingly, the problem also occurs when using 2.7 [5fdd53120e] on both the client and the server! This is most disturbing.

I wonder if there's something corrupt in the sqlite.fossil on the server. Yet, I'm able to log directly into the server and open it without issue. When run on the server, test-integrity takes over five minutes, having to chew through 79182 artifacts rather than just the relative handful that made it (corrupted?) into the clone. Here's the output:

skip phantom 73940 f04ded1d9b40d54463162264e37e6d92411d09427eea592ef05681035e2f2e64
skip phantom 73941 9dd591ef24b302a5fe2af0619d0cda6733348bacc541b3c0a134ac25981d4b2a
skip phantom 73942 5981969cad5afebb8f8098489d0419cc9ead560a0f22a484230f1886011cd57c
skip phantom 75591 469694857a5301e1da8cf067017d400403f3b12882d4265cefb3fd185c536305
skip phantom 75592 db32506880e727597e8629c21d29a4b5e3360bec7548d98ecdbb09685fd6e7a8
skip phantom 78638 caee77d9a8e1da2551cef44ba7a11e2de36eb8c047273148b4126bfc1edd21da
skip phantom 79127 d103c041ccb3a009926b6aa34a283a7cb8e4a645711ecd7a3002a90558d02e9d
skip phantom 79128 a3e9e62c6cafac5b5f103edcff77085b463085fca60c515aa16723c3d0006d44
skip phantom 79129 dcac190277fab9c6a9bdf651d18b25d97e0db49b3cc709ca145403c8b556bce3
skip phantom 79130 8e59dc7807d15d4fca6ee7b6a624447748eb24afb48f902fca27c2d550d84f1c
skip phantom 79131 4889866915b8b8fc28ca20cfc3f2ed4890eb1f0712b0537298fbdd498ce41510
skip phantom 79132 43c3d2d707a7c54dcb75660be9347d6bd79c4844f81f6ed73446570876f5f72b
skip phantom 79133 4b575b8050cdb703df26d28228a60b88f60e5279453ba71c65e0739575609a0b
79169 non-phantom blobs (out of 79182 total) checked:  0 errors
low-level database integrity-check: ok

Huh, by the way, I just noticed MSIE11 automatically corrects the capitalization of "sqlite" to "SQLite". It would appear SQLite has become important to Microsoft.

By andygoth on 2018-09-26 04:44:14 [link]

More information:

If I scp the repository to the target machine and then clone it from one file to another, it's fine.

If I run a server process on the target machine hosting the scp'ed copy then attempt to clone from that via http://localhost:8080/, it fails.

If I rebuild the repository on the server then clone again to the target machine via HTTP, it still fails.

If I log into the server and clone the repository file to another, it's fine, just like when I do this on the target machine.

If I log into the server and clone the repository HTTP URL to a file, it fails.

In all cases, having the HTTP server in the middle seems to be the cause of failure. The clone operation gets 13100 artifacts then stops.

Also curious: the repository file is 198 megabytes, a successful clone is 84 megabytes, and an unsuccessful clone is 20 megabytes.

By jungleboogie on 2018-09-26 05:00:07 [link]

I can't clone:
$ fossil clone https://andy.junkdrome.org/misc/sqlite.fossil andy.fossil 
server says: 404 Not Found
Clone done, sent: 279  received: 509  ip: 107.191.98.79
server returned an error - clone aborted

I can download, but the file stops at the ~20MB mark:
$ ftp https://andy.junkdrome.org/misc/sqlite.fossil
Trying 107.191.98.79...
Requesting https://andy.junkdrome.org/misc/sqlite.fossil
100% |******************************************************************************************************************************************************| 20096 KB    00:49    
20578304 bytes received in 49.35 seconds (407.20 KB/s)

By andygoth on 2018-09-26 05:21:11 [link]

I'm not running a Fossil server on andy.junkdrome.org, just hosting the repository file per drh's request.

As for stopping at the 20 megabyte mark, that's the whole problem. That's as much as I get when I try to clone from the computer that's actually running the Fossil server within my internal network, which I can't expose.

By drh on 2018-09-26 12:57:21 [link]

What webserver are you using? Is it somehow configured to limit the size of HTTP replies? Are there rewriting rules? Have you tried cloning using https: instead of http: to help ensure that nobody in the middle is corrupting the data coming over the wire?

By andygoth on 2018-09-26 16:20:56 [link]

I'm using Fossil's own web server, that's all. No other web server is running.

I have the following in my /etc/inetd.conf:

80 stream tcp nowait.1000 root /home/fossil/fossil/fossil /home/fossil/fossil/fossil http -host SERVERNAME -repolist /home/fossil

Also, /home/fossil/fossil/fossil is hilarious. :^) The username is fossil, the binary is in the fossil subdirectory, and the binary is called fossil.