Fossil Forum

Phantom object exists but not shown in UI
Login

Phantom object exists but not shown in UI

(1) By Dan Shearer (danshearer) on 2020-09-09 09:02:09 [link] [source]

I have a case where the UI is silent about a phantom object shown at the commandline. I don't know how much phantom objects matter since this is my first one :-) but it does at least seem inconsistent.

I have a repo with one phantom object:

me@example:~$ fossil test-integrity --parse example.fossil 
skip phantom 1668 7a9aa21c3506a10ab9465540e81071b39bca447d
1667 non-phantom blobs (out of 1668 total) checked:  0 errors
217 total control artifacts
  215 manifests
  2 clusters
low-level database integrity-check: ok

This agrees with "fossil git export" saying "missing check-in: 7a9aa21c3506a10ab9465540e81071b39bca447d" , which is how I noticed this phantom object in the first place.

However, the UI refuses to show me this phantom object.

https://example.org:1234/scm.cgi/example/phantoms gives "Page not found".

and

https://hueso.shearer.org:1234/scm.cgi/example/bloblist?phan

Just shows me ordinary objects with no reference to phantoms anywhere, not even the menu item that I notice is present when I do the same thing on the SQLite repo.

Dan

(2.1) By MBL (RoboManni) on 2020-09-09 16:22:45 edited from 2.0 in reply to 1 [link] [source]

did you try with Fossil Version 20-09-06 14:49:24 5c90832d79 ?

  1. http://ip/stat
  2. Artifacts
  3. Phantoms

or directly

  • http://localhost/bloblist?phan

(3) By Stephan Beal (stephan) on 2020-09-09 16:42:57 in reply to 1 [link] [source]

I have a case where the UI is silent about a phantom object shown at the commandline. I don't know how much phantom objects matter since this is my first one :-) but it does at least seem inconsistent.

A phantom is a hash which fossil knows exists but for which it does not have any content. It's exceedingly rare that a user would actually stumble across one in day-to-day use (i don't recall ever having done so in more than 12.5 years).

What you're seeing certainly seems like an inconsistency but possibly (can't say for sure) an intended one except for bloblist?phan. Looking at the code, that one certainly "should" be showing phantoms in the last column of the table, and does so on the core repo:

https://fossil-scm.org/fossil/bloblist?phan

You can see the whole list of phantoms with:

fossil sqlite
SQLite version 3.34.0 2020-08-26 10:50:48
Enter ".help" for usage hints.
sqlite> select * from phantom;

The result is the list of blob.rid values. You can get the uuids with something like:

sqlite> select uuid from blob where rid in phantom;

(4) By Dan Shearer (danshearer) on 2020-09-10 14:05:46 in reply to 3 [link] [source]

Stephan,

A phantom is a hash which fossil knows exists but for which it does not have any content. It's exceedingly rare that a user would actually stumble across one in day-to-day use (i don't recall ever having done so in more than 12.5 years).

Ok, if it's that rare then we go for root cause first and never mind any other chaos it may cause (which is more than just UI issues.) How did I create this phantom, apart from my impressive knack for breaking things?

I can replicate it every time with both Fossil 2.10 and Fossil 2.13 (compiled ten minutes ago) as follows:

$ git clone https://example.com/git-repo
$ cd git-repo
$ git fast-export --all | fossil import ../example.fossil 
$ fossil test-integrity --parse ../example.fossil 
skip phantom 1661 7a9aa21c3506a10ab9465540e81071b39bca447d
1662 non-phantom blobs (out of 1663 total) checked:  0 errors
217 total control artifacts
  215 manifests
  2 clusters
low-level database integrity-check: ok

In case other people are also initially confused like I was: "fossil import" is the correct command to use, despite "fossil export" being deprecated in favour of "fossil git export"

Also, although I happen to be blocked by this rare inconsistency, Fossil git interoperability seems excellent in all other tests I've run, so thankyou :-)

(5) By Stephan Beal (stephan) on 2020-09-10 14:15:58 in reply to 4 [link] [source]

How did I create this phantom, apart from my impressive knack for breaking things?

You're using git import/export, which is 100% alien territory for me :/. i cannot even speculate what might go on in those bits or what "normal" is.

A phantom in itself is not a problem, but yours seems odd to me (but might be completely expected within the realm of git conversion - no idea).

(6) By Dan Shearer (danshearer) on 2020-09-10 19:34:44 in reply to 1 [link] [source]

This agrees with "fossil git export" saying "missing check-in: 7a9aa21c3506a10ab9465540e81071b39bca447d" , which is how I noticed this phantom object in the first place.

Turns out this is a cute wee fossil bug. If you have a file in your git repo called "manifest" which you then import to Fossil, Fossil creates a phantom.

To reproduce, paste these commands:

git init /tmp/git-test
cd /tmp/git-test
cat >manifest <<EOF
C Version\s3.7.17
D 2013-05-20T00:56:22.515
P 7a9aa21c3506a10ab9465540e81071b39bca447d
R c103e20e6479251bb5b95c9b2e375810
U drh
Z 0c97c4a397f8b004f5d6f5a8876f5f7a
EOF
git add manifest
git commit -m 'Testing fossil import'
git fast-export --all | fossil import /tmp/fossil-test
fossil test-integrity --parse /tmp/fossil-test

You should get the result:

skip phantom 3 7a9aa21c3506a10ab9465540e81071b39bca447d
2 non-phantom blobs (out of 3 total) checked:  0 errors
2 total control artifacts
   2 manifests
 low-level database integrity-check: ok

I didn't come up with this, but it works. There are 25,277,662 files called "manifest" on Github alone, so that's a lot of potential ghosts :-)

Dan

(7) By Stephan Beal (stephan) on 2020-09-10 19:56:25 in reply to 6 [link] [source]

Turns out this is a cute wee fossil bug. If you have a file in your git repo called "manifest" which you then import to Fossil, Fossil creates a phantom.

That's actually not a bug, though the existence of a phantom breaking your use case may be. Fossil's First Law of Structural Artifacts is "if it quacks like a structural artifact, it is a structural artifact," where "quacks like" means "parses successfully as a structural artifact."

From the docs:

https://fossil-scm.org/fossil/doc/trunk/www/fileformat.wiki

2nd paragraph:

The global state of a fossil repository is an unordered set of artifacts. An artifact might be a source code file, the text of a wiki page, part of a trouble ticket, a description of a check-in including all the files in that check-in with the check-in comment and so forth. Artifacts are broadly grouped into two types: content artifacts and structural artifacts. Content artifacts are the raw project source-code files that are checked into the repository. Structural artifacts have special formatting rules and are used to show the relationships between other artifacts in the repository. It is possible for an artifact to be both a structure artifact and a content artifact, though this is rare.

(emphasis added)

That last part is what you've demonstrated. The file's name is actually unimportant for this purpose, only its content is. You'll get (unless i'm sorely mistaken (which can't be ruled out)) the same effect if you call it "foo".

When a repo is deconstructed and reconstructed[^1], fossil doesn't know which of the blobs are what we colloquially call artifacts (what the above calls Structural Artifact). It simply tries to parse them all as artifacts, and those which quack like artifact are treated like artifacts. Those which aren't are "content artifacts" (opaque blobs). Your demonstration file quacks like one, so it is one. The phantom, in your demo case, is the artifact referenced by the "P" card, as fossil doesn't have a corresponding blob for that.

The structural artifact parser (manifest.c:manifest_parse()) will fail immediately, or very closely to it, if given a non-manifest to parse, as its file format is extremely rigid. In all likelihood, parsing will fail on the very first line in all but the most oddball of cases (and the rest will fail on the 2nd in all likelihood). Thus the "does this random thing parse like a manifest?" test isn't all that expensive to do.

Anyway... that's your dose of obscure fossil trivia for tonight.

[^1] = the deconstruct/reconstruct commands are not something end users need to worry with unless they're curious. They have a couple oddball uses, like creating empty repos or multi-root repos using reconstruct, and can, if one is careful, be used to selectively rip apart content for inclusion into/exclusion from a reconstructed repo. By and large, though, they're not commands most users ever need try out.

(8) By Dan Shearer (danshearer) on 2020-09-11 00:27:00 in reply to 7 [link] [source]

Stephan said:

The file's name is actually unimportant for this purpose, only its content is. You'll get (unless i'm sorely mistaken (which can't be ruled out)) the same effect if you call it "foo".

You are quite correct.

Still considering the rest.

Dan

(9) By Dan Shearer (danshearer) on 2020-09-11 02:38:42 in reply to 7 [link] [source]

When a repo is deconstructed and reconstructed [...] It simply tries to parse them all as artifacts, and those which quack like artifact are treated like artifacts. Those which aren't are "content artifacts" (opaque blobs). Your demonstration file quacks like one, so it is one. The phantom, in your demo case, is the artifact referenced by the "P" card, as fossil doesn't have a corresponding blob for that.

I looked at manifest.c and what you say makes sense. First I removed the newline at then end of my quacking manifest (since that is the first exit point in manifest_parse()) and sure enough, no phantom because it no longer quacks.

Then I modified manifest_parse() to be a NOP, and that also worked fine. I can import large git repos with no phantoms and seemingly everything else working.

Which then brings up a policy question: does Fossil want to support recursive repos hosted in git? It's not clear that these will necessarily work anyway, but Fossil is certainly trying at the moment.

It would be much easier and probably more sensible to say that Fossil doesn't try to look for artifacts when doing an import. Or at least have a switch like this so you have to explicitly ask for it:

  fossil import --detect-fossil-artifacts 

Dan

(10) By Richard Hipp (drh) on 2020-09-11 02:59:39 in reply to 9 [link] [source]

fossil import --detect-fossil-artifacts

That's a reasonable idea. The problem is what to do after they are detected? We can't just discard them, because they are part of the Git tree, and need to be in the reexport, or Git will complain. (Right?) So I think we will need to create a mechanism in Fossil that says something like "even though this artifact looks like a manifest for a check-in, treat it like an ordinary data file."

It is not sufficient for just the "import" command to ignore the special format of the artifact. The artifact will be reprocessed the first time you "push" or run "rebuild". We somehow have to make a permanent record of the fact that this artifact is pure data. There are lots of corner-cases to consider.

(11) By Dan Shearer (danshearer) on 2020-09-11 09:06:48 in reply to 10 [link] [source]

It is not sufficient for just the "import" command to ignore the special format of the artifact. The artifact will be reprocessed the first time you "push" or run "rebuild". We somehow have to make a permanent record of the fact that this artifact is pure data. There are lots of corner-cases to consider.

I don't know Fossil enough to know the corner cases, but the limits of these corner cases will be the pathological case of deliberate exploitation.

It will be possible to carefully construct Fossil artifacts in a git tree to do many unexpected things, for example a tree that contradicts the tree described by git. It will then become whack-a-mole to keep fixing yet another obscure interpretation issue in fossil import. (You also made the point that this problem persists after import, but I haven't got that far yet.)

Some good news: the format docs for git-fast-import say With the exception of raw file data (which Git does not interpret)... which suggests there is no designed-in possibility of artfully hiding git objects in Fossil. So at least we can't round-trip this Polyglot SCM) .

Dan

(12) By Dan Shearer (danshearer) on 2020-09-11 10:56:54 in reply to 10 [link] [source]

There are lots of corner-cases to consider.

There is also a canonical test case:

cd /tmp
git clone https://github.com/drhsqlite/fossil-mirror
cd fossil-mirror
git fast-export --all | fossil import /tmp/fossil-mirror.fossil
fossil test-integrity --parse /tmp/fossil-mirror.fossil

gives 11103 phantom objects, which I can construe to be saying Fossil isn't self-hosting in this respect.

And also:

manifest_parse failed for 72d6a2f4347dc44bff115b132d63e0ebeb36bd5cb63bac9928a018ff7ad43dae:
line 1: invalid source on A-card 
"A Fast Method for Identifying Plain Text Files"

Well the method worked, I guess.

Dan

(13) By Dan Shearer (danshearer) on 2020-09-11 11:35:08 in reply to 1 [link] [source]

After lots of debugging (thanks everyone) we seem to know what the problem is and I have a workaround for my particular case. In case the workaround is useful to others, here it is.

Root cause: Fossil considers all imported files as potentially a Fossil artifact (ie metadata). So a file that happens to look like a valid artifact is treated as one, even if the metadata it contains doesn't make any sense.

My specific problem: I have one single file in my git tree that was imported from a tree that was imported from a tree. In that long-ago tree, this file was a genuine Fossil artifact called 'manifest'. This file gets imported to Fossil, creates a phantom object and seems to cause ongoing troubles.

Workaround: Use the latest git history-rewriting tool to remove all reference to this file in all of history. Deleting history in an SCM is about the worst thing possible and has been particularly difficult in git, but the following worked for me. git-filter-branch as shipped with git has been officially replaced by git-filter-repo . Download git-filter-repo and:

git-filter-repo --path DIRECTORY/ANNOYING-FILE --preserve-commit-hashes --invert-paths

That stopped my phantom pains.

Dan

(14) By Richard Hipp (drh) on 2020-09-11 15:47:32 in reply to 13 [link] [source]

Thank you helping to track this problem down. I have now checked in changes to Fossil that should largely prevent stray Fossil control artifacts from being introduced into the repository using a "fossil import --git". In the new code, if an input from Git looks like a Fossil control artifact, a single comment-like line is added to the end. These prevents Fossil from thinking the file is a control artifact.

If possible, please use the latest Fossil and let us know if it works any better for you.

(15) By Stephan Beal (stephan) on 2020-09-11 16:11:30 in reply to 14 [link] [source]

In the new code, if an input from Git looks like a Fossil control artifact, a single comment-like line is added to the end. These prevents Fossil from thinking the file is a control artifact.

Trivia for those not familiar with manifest_parse(): adding garbage to the end of the manifest is, counter-intuitively, a computationally fast way to get the parser to fail (i.e. to say "this does not quack like a structural artifact") because the first "real work" which manifest_parse() does is check the "Z-card", which is at the end of all legal artifacts. Also computationally fast would be to prefix a non-upper-case alpha character at the start of the artifact, but appending it is cheaper/faster for fossil.

More trivia: the term "manifest" is largely historical. "Manifest" currently refers only to checkin artifacts, but manifests were the very first type of "structural artifact," so the code was named manifest_parse(), though it soon evolved to parse all sorts of structural artifacts.

Reminder to self and Richard: the Z-card verification is ignored if the input has no Z-card, which means that it will fail a bit later for non-artifacts. i.e. it permits artifacts to have no Z-card so long as the rest is parsable (but if it has a Z-card, it is required to be valid). All artifact types require a Z-card, though (perhaps that was not always the case?), so we could speed that up by a tiny tick by classifying anything with no Z card as a non-artifact. (Unless, of course, there are age-old historical artifacts from a hypothetical(?) time when Z-cards were not required.)

(16) By Dan Shearer (danshearer) on 2020-09-13 09:08:50 in reply to 15 [link] [source]

Stephan Beal (stephan) on 2020-09-11 16:11:30:

Trivia for those not familiar with manifest_parse(): adding garbage to the end of the manifest is, counter-intuitively, a computationally fast way to get the parser to fail (i.e. to say "this does not quack like a structural artifact") because the first "real work" which manifest_parse() does is check the "Z-card", which is at the end of all legal artifacts. Also computationally fast would be to prefix a non-upper-case alpha character at the start of the artifact, but appending it is cheaper/faster for fossil.

Even cheaper/faster is to fail the very first validity test in manifest_parse(), which is that the file ends in \n. To make sure there is no trailing \n on FILENAME we can add a space with no newline like this:

echo -n " " >> FILENAME

Or, if we blindly assume that FILENAME has a trailing \n like most Unix files, instead of adding data to the file we can just remove (what we hope is) the \n:

truncate -s -1 FILENAME

Dan

(17) By Dan Shearer (danshearer) on 2020-09-13 10:37:01 in reply to 14 [source]

Richard Hipp (drh) on 2020-09-11 15:47:32 said:

Thank you helping to track this problem down. I have now checked in changes to Fossil that should largely prevent stray Fossil control artifacts from being introduced into the repository using a "fossil import --git". In the new code, if an input from Git looks like a Fossil control artifact, a single comment-like line is added to the end. These prevents Fossil from thinking the file is a control artifact.

This appears to have fixed the import issue in that potential artifacts are detected and flagged with a comment. However this seems to have introduced other two other problem. The first might be to do with retaining an rid calculated before sterilize_manifest() appends a line rather than afterwards, although I haven't seen code that does this. In any case you can see the error on opening the repo by running the following shell commands:

NEWFOSSIL=/tmp/newfossilbinary
git init /tmp/git-test
cd /tmp/git-test
cat >manifest <<EOF
C Version\s3.7.17
D 2013-05-20T00:56:22.515
P 7a9aa21c3506a10ab9465540e81071b39bca447d
R c103e20e6479251bb5b95c9b2e375810
U drh
Z 0c97c4a397f8b004f5d6f5a8876f5f7a
EOF
git add manifest
git commit -m 'Testing fossil import'
git fast-export --all | $NEWFOSSIL import /tmp/fossil-test
$NEWFOSSIL test-integrity --parse /tmp/fossil-test
mkdir /tmp/fossil-test-open
cd /tmp/fossil-test-open
$NEWFOSSIL open ../fossil-test
$NEWFOSSIL close
$NEWFOSSIL version

I get the following, with an expected error of "bad card type" but an unexpected error on repo close:

Initialised empty Git repository in /tmp/git-test/.git/
[master (root-commit) ffdb193] Testing fossil import
 1 file changed, 6 insertions(+)
 create mode 100644 manifest
Rebuilding repository meta-data...
  100.0% complete...
Vacuuming... ok
project-id: 666d6d939eb776839f225b1143427d08c1606ca1
server-id:  ce66784e9431fac876bd82024d382c566d7c6414
admin-user: bert (password is "R2AcajbmkS")
manifest_parse failed for d11f1d1a7e3e1c6feda8559e1642a993df980f0ed5d93a4288714fc684def3bc:
line 7: bad card type
2 non-phantom blobs (out of 2 total) checked:  0 errors
1 total control artifacts
  1 manifests
low-level database integrity-check: ok
manifest
project-name: <unnamed>
repository:   /tmp/fossil-test
local-root:   /tmp/fossil-test-open/
config-db:    /home/bert/.fossil
project-code: 666d6d939eb776839f225b1143427d08c1606ca1
checkout:     d93694a0e4c75754425388795d80f1a30682f383 2020-09-13 10:00:21 UTC
tags:         trunk
comment:      Testing fossil import (user: bert@example.org)
check-ins:    1
WARNING: The repository database has been replaced by a clone.
Bisect history and undo have been lost.
This is fossil version 2.13 [18dee26eee] 2020-09-13 01:15:51 UTC

The second error is easy to provoke: do an import of https://github.com/drhsqlite/fossil-mirror and it segfaults around the year 2012.

On a trivial note I think the comment added by sterilize_manifest() is misleading:

Remove this line to create a well-formed Fossil control artifact.

That comment is only correct at the time of import. Anything could happen to that file afterwards, invalidating this comment and potentially causing confusion. Maybe something like:

# When imported from git, this file was detected to be a well-formed Fossil artifact. Adding this comment invalidated it so it would not be interpreted by Fossil.

Dan

(18) By Richard Hipp (drh) on 2020-09-16 13:23:44 in reply to 17 [link] [source]

Please try again using the the most recent check-in as of this post or later. I think the issue may now be resolved.

(19) By Dan Shearer (danshearer) on 2020-09-16 17:40:36 in reply to 18 [link] [source]

Please try again using the the most recent check-in as of this post or later. I think the issue may now be resolved.

As far as round-trip git interoperability goes (which this thread was about despite the subject line):

  • The original issue of phantoms remains fixed. My initial test case was a single phantom from a partial sqlite git tree, but it ballooned out into thousands and tens of thousands with a full sqlite tree, or postgresql, or fossil. I have re-tested these cases. No phantoms.

  • The secondary rid/fingerprint issue is fixed by this most recent check-in. So that means an imported git tree can now be opened in fossil without having inconsistency. And it can then be re-exported to git. I haven't done lots of circuits yet, but it does appear to be working.

  • a full sqlite tree can now be imported from the github mirror without segfaulting, then re-exported to git. So can a fossil tree, but importing a github postgres mirror still segfaults straight away.

Thanks for that.

Dan