fossil reconstruct: sha1 ↔ sha3

(1) By Florian Balmer (florian.balmer) on 2019-01-12 08:49:43 [source]

For repositories with an initial hash-policy of sha1, and a later switch to sha3 (which may be a common case for long-standing repositories), the fossil reconstruct command does not create proper manifest linkages, resulting in a /timeline view with "isolated" check-ins, i.e. without connecting rails to their parents.

Test script:

fossil init "sample.fossil" --sha1
fossil open "sample.fossil"
fossil ci -m "check-in sha1 0" --allow-empty
fossil ci -m "check-in sha1 1" --allow-empty
fossil hash-policy sha3
fossil ci -m "check-in sha3 0" --allow-empty
fossil ci -m "check-in sha3 1" --allow-empty
md "sample-data" & rem: use mkdir on *nix
fossil deconstruct "sample-data"
fossil close
fossil reconstruct "sample-reconstructed.fossil" "sample-data"

Schematic /timeline view for sample.fossil:

○ check-in sha3 1
│
○ check-in sha3 0
│
○ check-in sha1 1
│
○ check-in sha1 0
│
○ initial empty check-in

Schematic /timeline view for sample-reconstructed.fossil:

   ○ check-in sha3 1
   │
   ○ check-in sha3 0

  ○ check-in sha1 1

 ○ check-in sha1 0

○ initial empty check-in

Moreover, the list of artifacts for the reconstructed repository contains additional entries classified as "unknown", that are also listed by the fossil test-missing and fossil test-orphans commands.

(2) By Florian Balmer (florian.balmer) on 2019-02-05 15:35:29 in reply to 1 [link] [source]

I've done more tests (Windows 10, Ubuntu 18.04) with the reconstruct SHA3-256 enhancements.

TL;DR: Works fine. For my part, this branch is ready to be merged. Please let me know if there's any issues I need to address.

I've deconstructed and reconstructed a clone of the self-hosting Fossil repository with the current trunk version of Fossil, and with the version built from the reconstruct-sha3 branch (with the latest changes merged from trunk).

The deconstructed data from the self-hosting Fossil repository is really impressive, even more so given that the clone is only ~45 MB in size:

41'582 files
4'157'143'988 bytes

Reconstructing the self-hosting Fossil repository with the current trunk version of Fossil:

The batch script to compare the blobs reports 36768 artifacts that are not present in both the original and the reconstructed repository.
Opening the reconstructed repository aborts with repeated "content missing for FILE" error messages.

Reconstructing the self-hosting Fossil repository with the version of Fossil built from the reconstruct-sha3 branch:

The test-missing command reports the same artifacts for the original and the reconstructed repository: "24 missing or shunned references in 17681 control artifacts".
The batch script to compare the blobs reports 8 artifacts that are not present in both the original and the reconstructed repository. These artifacts are marked as SHUNNED in the original, and marked as MISSING in the reconstructed repository. If the shun table from the original is imported to the reconstructed repository, and the reconstructed repository is rebuilt, no differences are reported.
Opening and committing to the reconstructed repository works fine.