tar axvf fossil.tgz

i'm unable to reproduce that.

[stephan@nuc:~/fossil]$ f tar release x.tgz -R sqlite3.fsl 
[stephan@nuc:~/fossil]$ tar xzf x.tgz 
(no output)
[stephan@nuc:~/fossil]$ rm -fr SQLite_2022-03-26_135110_d33c709cc0

Can you give us more details, e.g. what platform and fossil version?

$ f v
This is fossil version 2.17 [f48180f2ff] 2021-10-09 14:43:10 UTC
$ uname -a
Linux archlinux 5.17.1-zen1-1-zen #1 ZEN SMP PREEMPT Mon, 28 Mar 2022 21:56:46 +0000 x86_64 GNU/Linux

I've replicated it using the latest fossil version 2.18 and with tip as well to the same effect. I also made a fresh clone of sqlite only to produce the same tarball.
I'm at a loss right now on this how this can be. I get two different sha1 of the release tarballs with each different version of fossil.
My tar is not broken because busybox tar and 7z also report the crc error. Interestingly the crc error only occurs randomly in between extracting anyone of the png files, but I'm not sure what this tells me.

$ f v
This is fossil version 2.17 [f48180f2ff] 2021-10-09 14:43:10 UT
$ fossil tar release sqlite.tgz
$ sha sqlite.tgz
a3dc5cad81de349f3faeba1455e7c24744dd4148  sqlite.tgz
$ ./fossil v
This is fossil version 2.18 [84f25d7eb1] 2022-02-23 13:22:20 UTC
$ fossil tar release sqlite.tgz
$ sha sqlite.tgz
feb62c845673e450bfaab7aa6f565f3102a958a9  sqlite.tgz

Doesn't repro on macOS 12.3.1, either with bsdtar or GNU tar. Tested with both the 2.18 release version and with version 2.19 [1bb4147fd2] 2022-03-28 08:34:25 UTC.

Also, regarding the fact that different versions of fossil produce different sha sums is interesting.

That's a known issue and has come up before in the sqlite forum. Any single byte of difference changes the hash, and nether fossil nor (AFAIK) zlib guaranty that they will always produce an identical byte stream for the same inputs. They only guaranty that what you put in is what you'll get out. How it's stored is their business and they're free to change it.

I wasn’t even bothering with hashes since the test input can also change, depending on when you last synced the repo in question.

It didn’t help that “sha” is ambiguous. By the length I assume it’s SHA-1, but…

Of course using a sha1 hash to compare anything must necessarily assume a static input, and if one exports a tar from a hash with fossil it should always produce the exact same files coming from a Fossil repository that hasn't been altered.

That would be ideal, but in practice we cannot guaranty that:

• It uses a 3rd-party library for the compression and that library is free to behave how it wants, so long as it does the job we pay it to do.
• Fossil is free to rearrange how it creates the zip files from one version to the next. Perhaps it changes their order or fiddles with timestamps or switches to upper-cases hashes for the top-most dir name, or whatever.

If we impose the restriction that it must produce the same hashcode for the same repo version, ignoring for a moment that we can't because of the first point, fossil would be required for forever generate tar/zip files in exactly the same way it does today, with no wiggle room whatsoever for further future improvements.

An example: we recently changed how the manifest files which get added to a zip/tar are added, eliminate the "#" line which "artificially invalidated" the manifest. (That invalidation was initially done to to avoid confusion if the zip's contents, including that manifest, were later re-imported into fossil.) Thus building a tar from a pre-Christmas fossil version will, with 100% likelihood, result in a different hash for the same project version. The hashes of the individual files within the tar are all recorded in that aforementioned manifest file, though, and they are 100% immutable.

Is your medium for viewing the posts not showing you everything? :-)

It's an admittedly bad habit of mine to stop reading and start replying at the first convenient point, and then forgetting entirely that there's still more to read :/.

file says it's a tarball

$ file sqlite.tgz
sqlite.tgz: gzip compressed data, last modified: Sat Mar 26 13:51:10 2022, max compression, original size modulo 2^32 107231744

But right now I'm going to absolve fossil of any blame and call it by saying something must be funky with my system because even if I test-tar the folder the crc-error still occurs.

$ f v
This is fossil version 2.18 [84f25d7eb1] 2022-02-23 13:22:20 UTC
$ f test-tar ../src.tgz *
$ tar axvf ../src.tgz
aclocal.m4
art
autoconf
compat
config.guess
config.h.in
config.sub
configure
configure.ac
contrib
doc
ext
.fossil-settings
.fslckout
install-sh
LICENSE.md
ltmain.sh
magic.txt
main.mk
Makefile.in
Makefile.linux-gcc
Makefile.msc
manifest

gzip: stdin: invalid compressed data--crc error
manifest.uuid
mkso.sh
mptest
README.md
spec.template
sqlite3.1
sqlite3.pc.in
sqlite.pc.in
src
test
tool
VERSION
vsixtest
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Previously it was complaining about the png files now its anywhere.

$ f test-tar ../vsixtest.tgz vsixtest/*
$ tar axvf ../vsixtest.tgz

gzip: stdin: invalid compressed data--crc error
vsixtest
vsixtest/App.xaml
vsixtest/App.xaml.cpp
vsixtest/App.xaml.h
vsixtest/Assets
vsixtest/MainPage.xaml
vsixtest/MainPage.xaml.cpp
vsixtest/MainPage.xaml.h
vsixtest/Package.appxmanifest
vsixtest/pch.cpp
vsixtest/pch.h
vsixtest/vsixtest.sln
vsixtest/vsixtest.tcl
vsixtest/vsixtest_TemporaryKey.pfx
vsixtest/vsixtest.vcxproj.data
vsixtest/vsixtest.vcxproj.filters
tar: Child returned status 1
tar: Error is not recoverable: exiting now

At my whits' end now.

But right now I'm going to absolve fossil of any blame and call it by saying something must be funky with my system...

i'm gonna go out on a limb and suggest that it might be bad RAM. That's what it sounds like to me.

I was wondering too, it seems to be to auto detect compression using file extension.

From manpage:

Compression options
       -a, --auto-compress
              Use  archive suffix to determine the compression program.

Does your fossil link against zlib-1.2.12? I think I'm facing the same problem.

In my case, I had the luxury of having fossil on another machine that still produced working tarballs and found out that the working and non-working tarballs are identical except for the gzip crc at the end.

I think this is caused by the fact that the crc is treated as a signed integer in gzip.c while zlib wants an unsigned one. The following patch to fossil fixes the issue for me (and doesn't break the behavior with zlib-1.2.11).

Index: src/gzip.c
==================================================================
diff --git a/var/tmp/gzip~orig.c b/home/luna/museum/fossil-scm/src/gzip.c
index 7f4ca36..730f1bc 100644
--- a/var/tmp/gzip~orig.c
+++ b/home/luna/museum/fossil-scm/src/gzip.c
@@ -31,7 +31,7 @@
 */
 struct gzip_state {
   int eState;           /* 0: idle   1: header  2: compressing */
-  int iCRC;             /* The checksum */
+  uLong iCRC;           /* The checksum */
   z_stream stream;      /* The working compressor */
   Blob out;             /* Results stored here */
 } gzip;
Index: src/zip.c
==================================================================
diff --git a/var/tmp/zip~orig.c b/home/luna/museum/fossil-scm/src/zip.c
index 9af5859..7612b5d 100644
--- a/var/tmp/zip~orig.c
+++ b/home/luna/museum/fossil-scm/src/zip.c
@@ -257,7 +257,7 @@ static void zip_add_file_to_zip(
   int nameLen;
   int toOut = 0;
   int iStart;
-  int iCRC = 0;
+  uLong iCRC = 0;
   int nByte = 0;
   int nByteCompr = 0;
   int nBlob;                 /* Size of the blob */

I think this is caused by the fact that the crc is treated as a signed integer in gzip.c while zlib wants an unsigned one.

Nice detective work :).

The following patch to fossil fixes the issue for me (and doesn't break the behavior with zlib-1.2.11).

An equivalent patch has been applied to the trunk. Please try it out and report any issues. "Works for me," but it also did before.

One detail about this bugs me, though: the sizeof(long) is guaranteed to be at least as large as sizeof(int) but is not required to be larger. Many people assume long is 64-bit, but that's very often not the case in C. The crc in question is written to the archive using only 32-bits, so on systems where long is 64-bits (like the one this message is being typed on) the crc will be truncated. Whether or not that will lead to a bug is unclear. It "works for me" but really should be tested on a wider variety of systems and repositories.

Edit: on the other hand, crc32() explicitly generates a 32-bit checksum, so it's presumably taking care of the case that sizeof(long)>4.

This fixes my problem. Thank you for the quick response.

I think the reason why zlib has a long there is that int is only guaranteed to be 16 bits long and if you want a data type that will be able to hold a 32 bit crc on every single platform that has a standards compliant C compiler, you have to use long (uint32_t might not be defined if the platform doesn't have a data type that is exactly 32 bits large (though like CHAR_BIT > 8, I've never seen that in practice)).

The problems come in if long is actually more than 32 bits large and a "negative" signed 32 bit number gets converted to it (0x80000000 becomes 0xffffffff80000000). What I do not understand is why it seemed to work with the previous zlib version (and why it still mostly worked with the new one if the buffer being crc'ed was not extremely small) but I've given up on trying to understand crc implementations a while ago :P

This indeed does fix my problem.
After going through hell stress testing my ram as suggested and buying a new ssd hard drive thinking this is some hardware issue, I'm relieved that it's not.
I can also confirm the unreleased commit upstream Correct incorrect inputs provided to the CRC functions fixes this issue without this patch.

The previous releases of zlib were not sensitive to incorrect CRC

inputs with bits set above the low 32. This commit restores that

behavior, so that applications with such bugs will continue to

operate as before.

However the commit message seems to imply that other apps are at fault, so is fossil the buggy one here?

so is fossil the buggy one here?

I would say so. crc32() wants you to pass the result of a previous crc32() call as the first parameter but the unsigned long -> int -> unsigned long conversion that previously happened changed the result to a value that crc32() would never return.

Consider this program

#include <stdio.h>

int main(int argc, char** argv) {
  (void) argc, (void) argv;
  printf("%016lx\n", (unsigned long) ((int) 0x7FFFFFFFUL));
  printf("%016lx\n", (unsigned long) ((int) 0xFFFFFFFFUL));
  return 0;
}

which prints the following on my machine

000000007fffffff
ffffffffffffffff

i.e. converting something with 32 bits from unsigned long to int and back to unsigned long works fine if it doesn't have bit 31 set but in cases where the bit is set, the higher 32 bits get filled with ones (but I think this is actually implementation defined).

so is fossil the buggy one here?

As Luna demonstrates, yes, but fossil's misbehavior was apparently hidden by what may possibly be described as a long-standing bug in zlib. When zlib was corrected, the bug in misbehaving clients was revealed. An unfortunate mixup for all involved parties.