I was about to say what Stephen said, but he beat me to it.

Just to be clear: Sebres, you do not have any up-to-date clones of your repository?

Perhaps we should accept this as a bug report with a resolution to enhance the documentation to encourage people to make use of the auto-sync feature to keep one (or more) remote clones that are always up-to-date, so that they do not rely on manual (and perhaps months-old) backups?

Also, is there something we can do to make it clearer that Fossil does not store complete project history in the local check-out, but only in the repository? I'm guessing the OP has had prior exposure to Git which does store complete project history in the local check-out, as Git does not have separate repositories and there is no other place to store the project history. With many people coming from a Git background, perhaps it behooves us to make this point more explicit.

I already answered to Stephen (see above), as I saw your answer.

No, it was not local repo (and no private branches, no local commits there, all revisions were pushed into remote repository). After restore, I get completely intact repository synchronized with remote-origin repository. So all SHA-IDs are the same.

Just the working copies lost the states.

An as already said, I don't know any other SCM, that will act here like a fossil.

Yes, if I understand correctly what you said. (Just I don't have _LOCAL_ database, I have only _FOSSIL_ file in working copy). Possibly a scenario I already provided describe it better.

I've no problem at all with holding of internal ref's as integer IDs, just searching for a way to "synchronize" both (WC & repo-file) or at least anything that can say me current state of the art (branch/commit-hash of WC).

Is there some command that I could use?

Anyway, I assume now I could get another issue: some of my repo's were exported to git using marks... So I suspect that the marks (of fossil) are broken now?! As for your statement with integer BLOB ids.
Am I right?

P.S. BTW, I get no messages per email saying me I get answers from you (does this forum send an email)?

I'd not want to subscribe all, I thought I could receive at least the answers to my comments/subject.

We deliberately do not do that, because there are robots on the internet that will locate services that do work that way and then sign up unsuspecting email victims to receive notifications in order to harass them. This is called "backscatter spam".

Just to be sure, we are talking about the same thing...

Following scenarion causes the bug I issued here:

   ## clone, backup and open:
   fossil clone https://.../test-repo /repos/test-repo.fossil
   cp /repos/test-repo.fossil /backup/test-repo.fossil
   cd /work/test
   fossil open /repos/test-repo.fossil

   ## change something & commit (possible multiple times) then push ...
   echo "change" >> a.txt
   fossil commit -m 'test'
   fossil push

   ## ... disaster case on /repos/ ...

   ## overwrite with backup & sync it with remote via pull
   cp -f /backup/test-repo.fossil /repos/test-repo.fossil
   cd /tmp/test && fossil open ...
   fossil pull
   
   ## check:
   cd /work/test
   fossil branch current
   ===> empty
   fossil status
   ===> all files are signaled as changed.

Oh, almost forgotten to amend - I was on windows (corporate pc).

If you read my post carefully, you will realize that I restored the "made between the time" revisions from remote origin (via pull).
All this revisions was pushed therein previously and have surely the same SHA-hash value.

So I hope you can explain me why this situation is unimaginable using any other SCM system of the world (e. g. git).
Anyway otherwise I do not really understand where is the problem to restore the states in case like this, if all IDs of the commits (SHA-hash) are unique...

Yeah. Perhaps reworking the checkout-database schema to store hashes instead of RIDs is going too far. For the near term, let's focus on reliably detecting the situation and giving a sensible warning. Perhaps some recovery mechanism can be worked out in the (rare) event that repository swap is detected.

Does re-opening the repository fix it?

If so, maybe all that's needed is for the checkout repo to have a UUID or similar that's specific to the repository it opened, so it can detect that the repository it was "opened" from has been swapped out from under it.

The only irreplaceable thing we'd lose that way is the stash, and we can warn about that.

The only irreplaceable thing we'd lose that way is the stash, and we can warn about that.

A warning at that point is informative but doesn't offer any assistance in avoiding loss of the stash, as there's no(?) recovery strategy which would allow them to keep the stash once RIDs are out of sync.

As soon a single RID is out of sync, any use of the checkout db (which includes the stash) risks what amounts to corruption unless fossil happens to recognize it (because an RID in the checkout is missing from the repo). The worst-case scenario is, IMO, that the RIDs are out of sync but the checkout's RIDs point to valid, but semantically different, blobs in the repo. "Cats and dogs living together."

Things you currently lose when the RIDs get renumbered:

• The stash (I didn't notice this before - but the STASH and STASHFILE tables also record RIDs from the repository.)

• History of all "merge", "add", "rm", and "mv" operations that have occurred since the last commit.

• Bisect history.

• The ability to "undo" the previous operation.

Perhaps some of the above can be remediated by storing additional information (example: storing hashes in addition to RIDs) and then fixing up the RIDs when a repository swap is detected.

(example: storing hashes in addition to RIDs) and then fixing up the RIDs when a repository swap is detected.

Exactly.

Additionally the storing would help also by possible manual repairs (if needed or no auto-repair possible for some reasons).

stash

If stash ls and stash cat also break when this happens, then I'd agree that re-opening isn't good enough.

I don't mind losing the record IDs of the stashes or being forced to push things back into the stash from patch files produced by stash cat, but being unable to extract the stash pieces before re-opening is a problem worth fixing.

undo

As with the stash, that's worth saving, as long as it's not difficult to achieve.

file operation history

I'm not considering that "irreplaceable" as it should be readily reconstructed from the shell history and the user's memory. I'm basing that on my philosophy of "Commit early, commit often," though, which I know not everyone shares.

bisect history

A bisect sequence isn't irreplaceable, in my view, since you'd presumably make the same good/bad decisions when restarting a bisect.

The window of vulnerability is pretty small here. You're multiplying the probability of a crash bad enough to require restoration from secondary-level backups by the probability of being in a bisect at the time by the average time you spend doing a bisect sequence.

This is one answer to Stephan's question above, why hasn't this problem been noticed more often before? Because it's rare and usually has only small practical impacts when it does happen.

I'm assuming above that the stash contains only textual diffs. If binary diffs in the stash are common enough, then even if stash cat still works without valid RIDs, that's still not good enough, since stash cat doesn't produce a format you can apply to a newly reopened checkout.

Personally, I doubt I've ever stashed a binary diff. There must be others that do that, though.

I'm surprised that I didn't have a disaster with this, myself, a couple of months ago when I replaced my SSD.

I have a RAID partition that is never backed up - because what it contains is stuff for which the recovery plan is 'the master copy of this is offsite or on other media, so I'll recover by recreating.' This includes movies, music, books, geodata, software distributions, ... and Fossil repositories. (For some stuff, like the geodata and the repos, I even maintain scripts that would govern the re-download process - make fresh clones, do a lot of wget's, etc.)

My working copies live on the SSD, and are backed up daily, and I've assumed incorrectly that they'd still be usable even if the repository were re-cloned. (The repositories are synced or at least pulled regularly, including by 'cron' jobs in case I forget to sync.)

I think I just was fortunate that I didn't have any uncommitted changes or stashes about. (I don't use the stash a lot - I tend just to make another working copy.)

Not being a regular here, I had no idea that local repos would have to be backed up. I had always presumed that fresh clones would suffice.

The impact of the disaster upon sebres is much greater than loss of working copies - the invalidation of rids in the workflow used for git mirroring (together with the fact that he has a great many branches divided among multiple git remotes) has apparently thrown his exports into turmoil, and I am not familiar enough with how that stuff works to advise him at all. (For what it's worth, several local working copies were successfully recovered with 'fossil close --force'/'fossil open --keep')

If you consistently sync with the same remote repo, probably your RID values will end up being in the same order, even if you reclone. There is no guarantee of that, but with a reasonable amount of luck, a reclone could be harmless.

As for the Git clones, I'm thinking it is only incremental cloning that will be messed up. To recover, you just run a full clone. That takes a while, but you only have to do it once, and then incremental cloning should start working again. I think. (Surely Sergey will let us know if I'm wrong.)

Surely Sergey will let us know if I'm wrong.

Sure, and he'll do :)

The main reason why I used export marks was not the performance (which is important also, but not primary role in my case).
The reasons is:

1. I've get experienced the situation that fossil export new history-tree for git already 3 times:
   • once as fossil changes the user-info as regards the mail (previously sebres <sebres>, then sebres@example.com <sebres>). Note, the mail and username are confused.
   • second time as it was corrected (so to sebres <sebres@example.com>), or it was something else I'm not sure anymore.
   • third time - as the format (normalization algorithm) of message has been changed again (from version 1.xx to some 2nd and then still one, if I'm not wrong again in 2.13 or nearby)... regarding the new-lines (and/or) leading/trailing spaces.
2. Each time the whole git-repository got new history-tree (with effort to me to find/rebuild/resync all branches and git working-copies, origins/remotes and sub-modules, etc).
3. As I switched it to export mark export (incremental) it saves for me and my colleagues a lot of work (in case fossil does again something wrong after update to next version), at least the history would be rewritten only from some common base (that is quasi frozen to me).
   Only one time I got it broken this way (do you still remember, Richard? ;) And this was anyway only part of whole tree (so I get it repaired many times faster).

And once the format of the export-marks has been changed (previous fossil version wrote only hashes, next wrote the rids too).

This all as regards the incremental export.

Additionally, my git-repo's are many time larger (commit/branch counts is 10x extensive as fossil-repo, because contains all the commits of fossil and 20+ years of corporate/private works), many different remotes (that should be then synchronized too), sub-modules, conflicts in foreign remotes and clones, several testing-, CI- & development-clones and so on.

So the export-marks really help me to avoid the complete disaster. If I'll try today to do the full export/import (without the marks), I'd produce totally new history-tree (of fossil) in all my git-repo's from begin of tcl-epoch, so I could start the search for another job (e. g. cemetery gardener - quiet customers, no stress...).

If you consistently sync with the same remote repo, probably your RID values will end up being in the same order, even if you reclone.

Not really. And I guess there are too many ways where it will be not the same:

• as already said, Jan has produced this once (with his wrong TZ), so other topology by new clone;
• if one does something wrong in local repo (without auto-sync), for example if wrong branch was merged and committed, and then removes (purge) this "failed" artefact (it is not the problem without auto-sync, even with pullonly also)... in this case the identity is increased once, so the next clone get rid equals old rid minus 1, starting from this artefact.
• in several other cases I can imagine if I take a look to the fossil source code.

If you consistently sync with the same remote repo, probably your RID values will end up being in the same order, even if you reclone. There is no guarantee of that, but with a reasonable amount of luck, a reclone could be harmless.

The rid's get out of sequence the first time a sync encounters both local and remote changes. If you occasionally work on a disconnected repo and then sync when you're back on the network, you'll eventually hit that case, and you likely will have rid sequence disagreements from that day forward. Or if you forget to pull just before committing, so that 'fossil commit && fossil sync' winds up both pushing and pulling changes.

Also, my cron jobs sync some of my repos with both core.tcl.tk and with mirrors on chiselapp, so I have some local repos that are routinely synced with multiple masters.

I'm still mildly surprised to have got away with the flawed plan of relying on a remote repo as the backup for a local one.

I'm still mildly surprised to have got away with the flawed plan of relying on a remote repo as the backup for a local one.

I don't see that as a flawed plan.

No checked-in work is ever lost due to this problem. You might have to be do a little finagling to salvage the edit that you were in the middle of in your checkout. The worst that might happen is that your uncommitted changes get so confused that you need to start over.

Do you regularly do remote backups of your checkouts, to preserve uncommitted changes in case your disk goes out prior to your next "fossil commit"? The worst-case scenario for the renumbering-RIDs problem is the same as losing your uncommitted changes in a check-out due to a disk problem.

The rid's get out of sequence definitely more often as assumed.

By my attempts to restore rids in FOSSIL DB (shash, current branch, etc) for several working copies, I found now another strange case.

For better analysis I've just restored yet older backup of tcl-core-repo (from Apr. 2017) and after pull I see for example following strange picture:

My test working copy (0-day vulni continuation work 8.5-based) contains a large stash that I would repair/restore...

This has a rid 163305 which points in the temp-fossil-clone to artifact [134e487b22] (committed by Jan, 2017-09-04 12:38:14).
And I found the real artifact where it should be (by the date of stash and known branch), so it was definitely [707127bd2d] (committed by me, 2018-08-30 11:08:51).
But in new temp fossil-clone (after pull) it got rid 153293!

sqlite> select * from artifact where hash='134e487b227864429a7c7ab99b77ed9aafb8690e' limit 1;
rid|rcvid|size|atype|srcid|hash|content
163305|254|577|1||134e487b227864429a7c7ab99b77ed9aafb8690e|
sqlite> select * from artifact where hash='707127bd2d5bf27523922cb854210ab4d1b56e187b51d298b89387bc5ce8a47c' limit 1;
rid|rcvid|size|atype|srcid|hash|content
153293|254|1535|1|152421|707127bd2d5bf27523922cb854210ab4d1b56e187b51d298b89387bc5ce8a47c|

I know that the topology (between Apr. 2017 and today) is newly built and may be therefore totally different.

But the point is: how it may be that the newest artifact has lower rid as older?

Still again:

707127bd2d, 2018-08-30 11:08:51 --> 153293
134e487b22, 2017-09-04 12:38:14 --> 163305

If the synchronization occurs branch-related or grouped somehow else, the order how the rids are created is mostly undefined.

And it looks so, here is an output of 5 last commits from this repo with the rid's ordered by commit-time - as one can see the rid's have different order here:

sqlite> SELECT datetime(e.mtime) date, e.objid rid,
   ...>   coalesce(e.euser,e.user) user,
   ...>   printf('%15s', (SELECT value FROM tagxref WHERE rid=e.objid AND tagid=8)) branch,
   ...>   quote(substr(coalesce(e.ecomment,e.comment),0,50)) comment
   ...>   FROM event e
   ...>   WHERE e.type='ci'
   ...>   ORDER BY e.mtime DESC limit 5;
date|rid|user|branch|comment
2019-01-15 12:34:27|166639|sebres|          trunk|'next amend to [3e0a2d99f3]: fixes TclGetIntForInd'
2019-01-15 11:38:22|166622|sebres|          trunk|'fixes out-of-range index [string is ... -fail idx'
2019-01-14 20:03:36|166651|sebres|          trunk|'merge 8.7 (mingw/win-autoconf build, etc)'
2019-01-14 19:59:19|166656|sebres|  core-8-branch|'merge 8.6, conflicts resolved in win/Makefile.in '
2019-01-14 19:51:02|166650|sebres|core-8-6-branch|'normalize package provide for tcltests 0.1 (decla'

And compare with the output for same statement of my fossil-clone, I used to do this commits:

2019-01-15 12:34:27|166706|sebres|          trunk|'next amend to [3e0a2d99f3]: fixes TclGetIntForInd'
2019-01-15 11:38:22|166703|sebres|          trunk|'fixes out-of-range index [string is ... -fail idx'
2019-01-14 20:03:36|166701|sebres|          trunk|'merge 8.7 (mingw/win-autoconf build, etc)'
2019-01-14 19:59:19|166697|sebres|  core-8-branch|'merge 8.6, conflicts resolved in win/Makefile.in '
2019-01-14 19:51:02|166691|sebres|core-8-6-branch|'normalize package provide for tcltests 0.1 (decla'

As one can see the rid's are ordered here (same sequence as they have been committed by me).

From this point, the rid's get out of sequence every time you don't pull a bit longer (so even a restore of few days older backup could produce totally different rid's sequence and related topology as it was before the possible disaster).

FWIW, Richard added a check for this mismatch condition to the trunk today, so it may be worth updating. There is not yet an automatic recovery, but with this addition fossil can recognize and report the problem.

And now try to read carefully the whole discussion (it is always good before one writes a comment).

Starting situation: you have a local (cloned) repository of some remote, that has several working copies checkout's with different branches, where you are actively working.

The issue is - nobody makes a regular backup of such cloned repo's, because distributed and so they could be anytime "restored" from remote/origin repository (using pull or new clone). And one expects the certain consistency for all distributed copies no matter which SCM/VCS's (like git, hg/mercurial, etc or even fossil) used.
So the theory.

The worst case is - if you have lost a repository-file for some reasons and restored it from a bit older backup (and it has been kosher synchronized hereafter via pull from remote), your hash/uuid's are consistent with the remote, but unfortunately not the rid's...

And this is actually not the problem (because it is really internal thing of fossil), but (as long as there is no possibility to repair/restore)...

This means that several working copies will:

• lose a references to current checkout, what makes impossible to find any modifications (which were accidentally or because WiP not yet committed) in all the working copies (do you still remember - you are actively working with several working copies with different branches etc);
• lose all stashes (because of wrong id of stash as well as id of the blob's involved in stash);
• if there are export-marks (that internally reference to rid's of commit/blob), the import (in other mirrored or joined repositories, like git) fill create a new history-tree (starting from the point where new rid-related topology was created);
• all other things are mentioned by Richard in /forum/forumpost/b5947863d2;

Additionally drh as well as some other already see this or at least inclined to consider this as a real issue.
I must therefore ask you to stop advising here what one "should not do" or "should expect"... Please believe me, I'm aware.

Additionally drh as well as some other already see this or at least inclined to consider this as a real issue.

Indeed. And I'm increasingly aware of the seriousness of the issue. I hope to find time to fix it soon. I have plans to do so. In fact, the latest trunk version of Fossil does at least detect when a repository has been swapped out for a clone that has different RID values, and it issues a warning when this happens. It just cannot fix the problem, yet. I would have taken care of this already, but I am struggling with more important problems on other projects right this minute, and this problem has been in Fossil for over a decade without causing serious inconvenience, and so I think it can probably wait a week or so. Thank you for your patience.

Rebuild your fossil executable using the latest code from trunk. The new detection logic is completely automatic. The limitation is that it currently just gives you a warning, rather than actually fixing the problem.

Please do so. Thanks.

Will this leave whatever information might be useful for recovery/forensics in the .fslckout/FOSSIL file around for use?

'close' removes the checkout db, so those would be lost. Forensics are unlikely, IMO, to be of any use because there's literally no reliable way to match up mismatched RIDs, so there's not much one could do with the raw data in the checkout db. The causes of the mismatch are well-understood, so an investigation of why it happens is unnecessary.

Seems like a last resort thing to recommend without investigating the potential loss and recovery options.

There are no recovery options until an auto-recover can be implemented, and there will be cases where auto-recovery cannot recover. In short, the checkout db needs to hold a map of RIDs to UIDs in order to be able to determine if a complete recovery is even possible. For any UIDs which the checkout has but cannot be mapped to the opened repo, recovery really isn't a possibility (IMHO). That could happen if, e.g., one recovers the repo from an old backup which doesn't contain all checkins which were made before the original repo db was "recovered".

Specifically, even though --keep is used, it most likely won't open to the correct checkout which means the files that are actually modified won't diff correctly.

They will diff correctly vis-a-vis the newly-opened checkout, which is as close as we can currently get. Worst case, the diff is larger than intended. Once a mismatch has occurred, because one or the other db was replaced, there no way to get them them diff "correctly" because the version they were intended to diff against is now "lost information".

For the case it may help by the searching of solution to restore export-marks:

I wrote a small helper script repair-export-marks.tcl that already has successfully restored the export-marks (the swapped rid's within).

Expects Tcl (tested with >= 8.5) and sqlite3 (I used 3.26.0)...

Result example for restoration of marks for my clone of tcl-core-repo:

  Repair fossil export-marks: .fossil2git-fssl -> .fossil2git-fssl.repair-export-marks-20190117
  Done. ** marks 113626 checkins 24004 blobs 89622 sane-rids 100668 wrap-rids 12958 wrap-checkins 3349 wrap-blobs 9609 **

Meant that 12958 of 113626 rid's got swapped in the fossil-marks file.

And the prime thing: Git import does not produce wrong check-in's and the whole history-tree is still consistent after import (across all git-refs: refs/remotes/fossil/, refs/remotes/upstream/, refs/remotes/sebres/*, etc).

@Richard, for the case you can use something from my work...

I made new script repair-local-wc.tcl: that could be used to analyze local working copy states as well as to try to repair/re-swap working copies (current checkout, stashes inclusive referenced files, etc).
The script has parameter "process" (default 0), if not set to 1/true will do analysis (test-run) only and both fossil databases are opened in readonly mode in this case.

I had basically no interest for undo's (really temporary state in my case, mostly to neglect already after few hours), but if it is needed, could be done similar to the stashes.

For an example output of the test-run of script (using generated map of export-marks), see "repair-local-wc-example.txt".

Additionally I've extended my other (mentioned above) script "repair-export-marks.tcl" to generate map (a tcl-dictionary with old/new) for RID's from export-marks. This can be used by restoring of local working copies, because faster and have more correct RID's on the fly...

Because I have the export-marks for almost all repo's (have to repair), I used this RID-map by analysis/repairing of my copies, so I actually don't know how good the script will process without.

Do not hesitate to contact me should you have any question on this scripts etc (E-Mail, Tcl-chatroom, github, whatever).

The issue is - nobody makes a regular backup of such cloned repo's, because distributed and so they could be anytime "restored" from remote/origin repository (using pull or new clone)

Really? What distributed guarantees is that you can get all the committed and synced objects from the remote source. If the remote was unavailable when you did a local commit (with autosync) or you forgot the manual sync that is part of your process, (or ...) you won't get it back. The repository you just lost, a new clone, and a restored old repository + pull, will all have different RIDs, and possibly other differences as well.

Do you back up all your working copies regularly? A working copy can not (even when the current issue is covered) be guaranteed to be independent of the repository it is based on, so you should be backing that up too.

Really.
And I never got situation like this using git, hg (even fewer "distributed" svn or cvs too) working on 3 different places and syncing repo's and working copies remotely.

Also note the emphasis was on word "regularly".

And I already wrote about fundamental case where even a 1 day backup/restore may cause swap the repo out.

I had days where I created 15 very large stashes testing a new feature and searching a bug, the half of that were important.

Savvy?

And yes, I do back up my system basically (but even rarely full-sync-able repo's including fossil). Until now with regard to the latter (and I'm still pretty sure about my git&hg-clones).

Let alone the backup of git/hg are pretty easy using the push method to remote which is backup, because it supports multiple remotes:

git push --all "git://my-backup-remotes/..."

That are and remain consistently as regards history-tree/topology/exports/etc of both remotes/clones.

This is the prime reason I thought fossil follows the same rules and therefore disregarded regular backups.

Exactly.

But I don't see a large problem to "repair" the situation in the future. As well as I can't understand the Stephan's argu's about a "impact memory usage and, to a lesser degree, performance, etc".

One should just store a RID to the last integer-ID in local-checkout DB additionally. And verify it (and swap others) only in case the RID of last checkout deviates from RID stored in the cloned-DB (so effectively the comparison takes place only once if fossil-app starts to process command).

One compare. Once. That's it.

If the checkout-database tables are modified to store 32-byte SHA3 hashes instead of 2-to-4 byte integers, that takes up additional memory. Extra CPU cycles are consumed in moving that memory around and in comparing the values.

If the RID values change in the repository, it is not possible in general to compute the mapping from old values into new values, so there is not a general way to update the check-out database. We can pull some tricks to remap some of the RID values in the checkout database, but not all of them. In order to do a full remap, the checkout database will need to be enhanced to store additional information that is only used when it becomes necessary to remap the RIDs.

If the checkout-database tables are modified to store 32-byte SHA3 hashes instead of 2-to-4 byte integers...

Sure, but this was not my suggestion. I would only provide additional map (rid <-> hash), similar way the export-marks. So not replace int's but complement.

If the RID values change in the repository, it is not possible in general to compute the mapping from old values into new values?

Why not? For example cloned repo had 123 <-> abcdef entry. Same map in local working copy DB. After possible swap (fossil can determine it rapidly once by start, simply using last commit hash-rid comparison), e. g. new entry would be 120 <-> abcdef, the cloned rid's (120) could be rewritten using local rid (123), or vice versa. There are not too many references in local DB, thus possible this way it could be implement faster and with lower effort.

Anyway the determine process is important, to notify user - there is something wrong. Further work to synchronize both could affect as an extra fossil command. At least the important thing could be retrieved and wrapped easy: current commit and stashes in work-copy... Other like export marks, etc. may be a to-do.

Why not?

Richard was referring to the current code/schema, where there is no checkout-side mapping of RIDs/hashes (checkouts currently use only the RIDs). In the current db scheme this reverse mapping cannot be done. It requires changes to the db schema and number of queries.

... or as already said simply extending a local-DB with new table mapped rid's <-> uuid's (by first usage of rid locally or by writing it to local-DB first time)
and the restoring process from there (if swap of the cloned repo determined), resulting in several statements like:

-- disable constraints (if some are there) --

update rid_uuid_map l set rid = (select rid from tmp_tab t where t.uuid = l.uuid);

update vvar set value = (
  select rid from tmp_tab t where t.uuid = (
    select value from vvar where name = 'checkout-uuid'
  )
) where name = 'checkout';

...

-- enable constraints --

and similar.

All internal references in local DB may retain further as integer rid's (as long as there is resolvable from map).

And I want just to amend: the primary role of each SCM (especially distributed version control systems) is to provide the consistency of the history tree to the outside.

Internal it could use what it wants, but to the outside it should always deliver the same data (no matter what outside actually is - the clones, origins/remotes, working copies or exports). Emphasis on always.

Normally, a rebase-process is only acceptable if the user want it explicitly (forces it), for example if local revisions (one want to rewrite/remove/rebase/amend) are still not yet synchronized with the remotes.

(i'm replying to the top of the thread because this response is aimed at several of the deeper-nested responses which seem to be exaggerating and/or misunderstanding the significance/relevance of this "RID mismatch" issue...)

Management summary: Your data are safe. Don't panic. Keep fossiling.

To clarify exactly how important RIDs (Record IDs) are to fossil's data integrity: not at all.

RIDs are, in essence, the primary key for an internal cache which is unique to each copy of a repository, and they have zero relevance for the long-term integrity of the underlying data. They are an internal optimization for linking records which themselves have no ID other than their unique hash. At its lowest level, fossil only references its data using their hashes, but RIDs are used "everywhere" internally because they're more efficient and easier to work with in C code. Fossil never, ever shares RIDs between two copies of a repository, nor does fossil publish any RIDs to the outside world via links or such. Each clone/copy uses only its own internal RIDs, and never compares/syncs RIDs with any other clone. Using the "deconstruct" and "reconstruct" commands will literally eliminate all RIDs and recreate the DB from scratch, which re-creates/re-assigns the RIDs (remember, they're just part of an internal caching mechanism).

Unlike a repository db, which is standalone and "durable", a checkout maintains its local state in a transient/disposable database which links its local state to the copy of the repo it was opened from, and it uses that copy's RIDs (artifact cache keys) to maintain that link. The relationship between repos and check databases is one-way: a checkout necessarily knows its repo but a repo does not know (nor need to know) about any of its checkouts.

This "RID mismatch" problem occurs only when one or both databases in a given repo/checkout pair is/are replaced with a copy which has a different sequence of RIDs. Except for one known hypothetical mismatch case which, as far as we know, has never happened in the wild, this problem is "cosmetic", in that it does not lead to data loss or parent/child mismatches in the data hierarchy, just confusion (as is evidenced by this thread). Closing and re-opening an affected checkout db is all that's required to resolve it (see the first response in this thread for the exact commands and an explanation of their relevance).

The fossil devs, above all Richard, are aware of the problem and it will be resolved at some point. Until then, it is most certainly not an "end of the world" situation and has never, in and of itself, caused data loss (though it's certainly possible that the resulting confusion has led developers down paths which caused collateral loss). As an interim workaround, Richard has added a feature which "fingerprints" a repo, so that such a mismatch can be detected and the user warned - simply update your fossil binary to the current trunk version to get that feature.

Agree. This is indeed not "end of the world" situation.

nor does fossil publish any RIDs to the outside world via links or such.

Well, this is not whole true - export-marks contains RID's, thus yes, it does indirectly. Which as already said could cause new history-tree on mirrors, which is very bad if checkins are referenced elsewhere (foreign repo's and merges, sub-modules, etc).

As well as the sentence - this problem is "cosmetic" - sorry, but it sounds a bit undervalued to me.
From the point of view, I've not directly lost the data at the moment (excepting several stashes, that are important to me and I still don't know state of 2 working copies, because still can't find the matching check-in, so dunno some local modifications in the working copies are present or not), let alone some other minimalist issues... But I've lost a lot of time (almost a half week) on this problem (repairing, searching, checking, syncing with mirrors, etc.)...

Well, this is not whole true - export-marks contains RID's, thus yes, it does indirectly. Which as already said could cause new history-tree on mirrors, which is very bad if checkins are referenced elsewhere (foreign repo's and merges, sub-modules, etc). ... As well as the sentence - this problem is "cosmetic" - sorry, but it sounds a bit undervalued to me.

RIDs are used in git incremental exports (apparently - i've never done a git export). As Richard mentions somewhere above, doing a full re-export resolves it. It's a "cosmetic" problem in the sense that no information is irrevocably lost, provided one doesn't lose all of their up-to-date repo copies/backups.

This may be incorrect, but i believe the "marks" used in incremental exports are not retained by git after it imports them. If it does then the git export is fundamentally flawed for using RIDs as a reference point. If git doesn't retain them then this is, IMO, a non-problem.

doing a full re-export resolves it.

No, as I already mentioned above, it does not. Full reexport will make it still worse, because creates totally new history to me... This is caused by the lot of incompatible changes in fossil export process during several years (committer name, messages etc), which cause new hash values.

After import in git you'll see 2 completely independent trees (timelines), where one (previous) tree referenced in all foreign remotes and merges, submodules etc, and new timeline (with new hashes) parallel show totally new tree (with no one revision referenced). And this (new) tree grows with next imports (during the old tree does not become continuation anymore).

The incremental export protected me even for situation like this. See hier for more details, which consequences it had to me.

Don't forget that some long life branches cannot be rebased (due to conflicts).

Anyway, the reexport is not an option in my case, so unfortunately it is not "cosmetic" issue to me.

Fortunately, I could few or more successfully resolve the export issues...

See - https://fossil-scm.org/forum/forumpost/6d53c630c3?t=c

Just one older repo still has an issue be repair and consequently by import. This goes to the confusion of two check-ins with pure file-blobs. Previously both RIDs have cNNNNN and bNNNNN marks. During repair only blob can be found (as well as the uuids references really only the files artifacts in fossil)... And I cannot find event entries for this RIDs (fit the uuids in artifacts). Strange, but...

I assume this repo has get some "broken" export (purge or something else in-between), so now it is a bit out of "tree".

Just lucky in the circumstances, exactly this repo has not too many foreign references, so if I don't find the solution to convince the git continue the import incrementally, I could rewrite the foreign branches in other remotes (using rebase, cherry-pick or squash) and update refs in submodules manually.

Thank you, Richard.

I've a little question about the branch - is the field "vfiles.mhash" really necessary? I mean, does the value of "checkout-hash" (now in "vvar" table) not enough for the simplest restoring of current vfiles-states of repo now?
Just curious, because I got it repaired (swapped the RID's of vfiles) via fossil checkout -f --keep $hash without this enhancement.

Simply to pursue an earlier Stephan's argument:

It would negatively impact memory usage and, to a lesser degree, performance.

The case the repo is in a temporary state of active merging/reverting process is almost to neglect here (either it is already committed, or the merge was "failing" due to many conflicts, or one could not complete this for some reasons) but such target of merge is apparent from the checkout/modifications.
Anyway checkout --keep does not do any changes on files of working copy, so all possible modification are still visible after the checkout (repair). So in the possible worse case, it is really affected, following scenario does the work too: backup local files -> re-merge/re-revert -> restore local files. Clear it expect manually activity, but IMHO it is anyway better in such (older) unfinished merge processes.

BTW, although the stashes are another thing, because they could really exist during considerable time and different stashes are based often on a different revisions.
But I believe also it could get the hash-field only in table stash, where the table stashfile can be repaired using RID's of base-checkout of the stash self (so table stashfile does not need to be altered with the new hash-field).

Just to hint the certain similarities:

stash(hash) <-> vvar(checkout-hash)
stashfile   <-> vfiles

Regards, Serg.

Sure.
Just my point was to ignore this too, also in case of single merge (because a bit hard to handle, and the case is rather very rare). Let alone, the merge-process is anyway repeatable as well as well apparent from the current checkout/modifications, after WC-checkout gets cleanly swapped.
But it's ok.

Just closing the loop: I have been using these changes all day, and they are now on trunk.