I set two non-empty files for hard move, and made one change in each, as well as a change in another file.

The result after commit is that both new files are empty.

Sequence of commands was:

$ fossil mv --hard troppo/pt.sls troppo/pieces.sls
$ edit troppo/pieces.sls
$ fossil mv --hard tests/pt.ss tests/pieces.ss
$ edit tests/pieces.ss

I happened to do a status before commit which correctly identified the rename (output edited to remove extra info):

EDITED     tests/pt.ss  ->  tests/pieces.ss
EDITED     troppo/buffer.sls
EDITED     troppo/pt.sls  ->  troppo/pieces.sls

Unusually, the commit took a while (i'm guessing a few minutes). I have autosync on with the remote set to my little raspberry pi server on the local network and there was no error code reported to the shell.

+ /usr/bin/fossil commit -v
Pull from http://paranor.thyme/germ/troppo
Round-trips: 1   Artifacts sent: 0  received: 0
Pull done, wire bytes sent: 388  received: 1326  remote: 192.168.31.254
ec ./ci-comment-7FF4C232CB9C.txt
Pull from http://paranor.thyme/germ/troppo
Round-trips: 1   Artifacts sent: 0  received: 0
Pull done, wire bytes sent: 390  received: 1326  remote: 192.168.31.254
New_Version: afedb5923b087b9b3c2eb206956d8743112598f93ab14dfee7732d363f27d32e
Sync with http://paranor.thyme/germ/troppo
Round-trips: 1   Artifacts sent: 3  received: 0
Sync done, wire bytes sent: 1819  received: 300  remote: 192.168.31.254

The version of fossil is identical on my development netbook (an old 32bit asus eeepc) and raspberry pi (model 1b+) server, which is the current package from Void Linux:

$ fossil ver
This is fossil version 2.23 [47362306a7] 2023-11-01 18:56:47 UTC

Both machines have plenty of hard-disk space.

If i click the commit hash through the web ui, i see this description:

Renamed and modified tests/pt.ss [5a43d90d4d] to tests/pieces.ss [a7ffc6f8bf].

with the entire file in red on the left-hand-side, but nothing on the right-hand-side.

Is there a good way to recover? If not, i can manually recreate the new files.

Also, it's a personal project so i can email the fossil repo if it's of any help. File size is about 3.3M and half of that is unversioned files which i could remove before emailing.

Thanks for reading this far. :)

i can email the fossil repo if it's of any help. File size is about 3.3M and half of that is unversioned files which i could remove before emailing

If you would please make a copy available, that would be very helpful. You can post a link to it here in the forum if you like, or email me a copy (stephan wanderinghorse net) and i'll make it available to the other developers via a non-public channel.

Are you trying to recover the new files including you most recent edits (not known to myself how to do this) or the old, non-empty versions from previous commits?

I'm guessing the first, but the second should be trivial, just to reassure.

I've emailed you the repository unchanged.

Thanks for looking into this.

The result after commit is that both new files are empty.

I used wrong/imprecise language here, by "new files" i meant the newly renamed files (each of which had one change as well). Sorry about that.

It would be good if my intended latest versions of those files were in the repository somewhere so i could restore them, otherwise it means that changes were lost during commit.

The changes included in the rename were only one line in each file so i can manually recreate from the previous versions of those files (which are still in the history), so from that sense, there's little harm done.

Just some more info.

ls -l --full-time in my workdir shows there was quite a delay between each file in the commit. I've pasted the ls output line for each file:

-rwxr-xr-x 1 jerry jerry 0 2024-06-17 12:42:18.829523270 +1000 pieces.ss
-rw-r--r-- 1 jerry jerry 5429 2024-06-17 12:36:00.812829327 +1000 buffer.sls
-rw-r--r-- 1 jerry jerry    0 2024-06-17 12:41:48.612467879 +1000 pieces.sls

I believe i started the commit command at about 12:38 and it finished at 12:42 according to my zsh prompts.

buffer.sls has the save timestamp from my editor session.

The other two files would've been modified by fossil.

buffer.sls has the save timestamp from my editor session.

Just FYI: fossil does not actually record timestamps of files. Instead, it treats all files as if they had a timestamp of the most recent checkin in which they were added or modified.

i started the commit command at about 12:38 and it finished at 12:42 according to my zsh prompts.

That's presumably running from a micro-SD card running on your pi1b. If you have the option, running your source trees from an old USB drive on that machine should considerably speed up fossil. Depending on the size of your source tree, you may also benefit from moving any swap space from the SD to your USB drive. To the best of my fallible recollection, no pi earlier than the pi4 (or 4b?) can boot from USB, but older pi models can mount external USB drives, which are invariably(?) faster (and more durable) than micro-SD cards. It's been my experience that an externally-powered USB hub is necessary for USB HDD drives on a pi1, but you might (might!) be able to run a USB SSD without its own power source.

Just FYI: fossil does not actually record timestamps of files. Instead, it treats all files as if they had a timestamp of the most recent checkin in which they were added or modified.

Just in case we're talking at cross purposes, those timestamps were from my workdir, ie. checkout directory on my eeepc netbook.

The buffer.sls timestamp was made by me from my editor session, and was untouched by fossil commit even though it had a change in the problem commit.

The other two timestamps were made by fossil during the move commit which i thought was interesting and wanted to mention in case it's helpful.

If you have the option, running your source trees from an old USB drive on that machine should considerably speed up fossil.

Thanks for the tip and it's something i do, in a way. I setup my raspberry pis so that boot/OS is run read-only from a micro-sd with read-write stuff like /home and /var mounted from a USB thumb drive.

My fossil repos are served from that /home dir.

It's something that's worked incredibly well for me - i've been running raspberry pis like this for over 10 years without any sd-card corruption - and i can power off, without shutdown, without harming the OS.

Anyway, i love those little devices: i get a fully functional and performant server (at least for my needs) all with a slightly less than 2W power draw.

And fossil is a perfect fit. :)

Oh, and the fossil service itself is run via scgi with a lighttpd instance acting as a reverse proxy.

Have you considered running under althttpd rather than lighttpd just to go fully down the rabbit hole?

😆

Is there a good way to recover? If not, i can manually recreate the new files.

While trying to reproduce this i stumbled over the repository having a pre-checkin hook script which (of course) isn't on my system. After doing fossil unset hooks, i am unable to reproduce this problem using the version prior to the tip (where the old filenames still exist).

Ergo... my suspicion is that the hook is either the culprit or is complicit in what's going on. Can you post the hook script named commit_no_xxx.ss?

For completeness's sake, here's a trimmed copy of my session, minus some of the noise and the config tweaks needed to get it working (noting that i was working from a direct copy of the original, not a clone):

$ f open ../troppo.fossil
..
$ f up prev # the version with the old file names
...
$ f mv --hard troppo/pt.sls troppo/pieces.sls
$ f mv --hard tests/pt.ss tests/pieces.ss
$ echo ';;' >> troppo/pieces.ss
$ echo ';;' >> tests/pieces.ss
$ f status
...
checkout:     a03fa7a8894e2e71693fe053264f9f05948ecd54 2024-06-17 02:34:39 UTC
parent:       361a350d8eec45516b7bd7e96eca9ce1b9edfab3 2024-06-15 10:57:52 UTC
child:        afedb5923b087b9b3c2eb206956d8743112598f9 2024-06-17 02:42:33 UTC
tags:         trunk
...
EDITED     tests/pt.ss  ->  tests/pieces.ss
EDITED     troppo/pt.sls  ->  troppo/pieces.sls
$ f diff
Index: tests/pieces.ss
==================================================================
...
+;;

Index: troppo/pieces.sls
==================================================================
...
+;;

$ f ci -m test1 -v --allow-fork
...
$ f ui # /timeline and diff view look good

That's a Chez Scheme script i wrote that checks for new lines containing XXX and installed as a before-commit hook that guards against commits for files that still have them left in.

(I mark comments with XXX as a note-to-self for things that need to be sorted out before committing.)

Apart from Chez Scheme, it needs irregex to work.

Just in case you haven't used Chez Scheme before, clone and add irregex to the list of library search paths. eg,

$ git clone https://github.com/ashinn/irregex /path/to/irregex
$ export CHEZSCHEMELIBDIRS=.:/path/to/irregex

I've recreated a diff of the changes that were lost, they didn't block the commit, or at least not all of it. :)

Index: tests/pieces.ss
==================================================================
--- tests/pieces.ss
+++ tests/pieces.ss
@@ -2,11 +2,11 @@
 
 (import
  (chezscheme)
  (testing)
  (only (toolbox list) flatten)
- (troppo pt))     ; module under test
+ (troppo pieces))     ; module under test
 
 (install-verbose-test-runner)
 
 (define pl1 (make-pieces))
 (define st1 "hello world")

Index: troppo/pieces.sls
==================================================================
--- troppo/pieces.sls
+++ troppo/pieces.sls
@@ -18,11 +18,11 @@
 ;; undo/redo operations intact.
 ;;
 ;; Piece tables are nicely described here:
 ;; https://www.catch22.net/tuts/neatpad/piece-chains/
 
-(library (troppo pt)
+(library (troppo pieces)
   (export
     (rename
       (list make-pieces))
     pieces-length
     pieces-string

BTW, i emailed you the client-side repo only, but yesterday i wanted to resume coding so i made a tar.gz of the checkout/workarea (with rubbish extra files and all), server repo, and client repo just in case. Compressed, that came to about 3.7M. I can send you that too if you think that'll help.

And here's the full before-commit script:

Edit: Script removed as it has the very serious bug for renamed-edits. The broken version should not be used by anyone.

I did look into at the time and even cloned its repository.

My memory is hazy, but i think the main reasons i went for lighttpd was that it supported scgi and there was a package for it under Void Linux.

I'm lazy and wanted someone else to track releases. :)

You're right Stephan, commit_no_xxx.ss is the culprit! Or at least its interaction with fossil.

I backed out the broken commit and tried again with the same bad result.

I disabled hooks as you did, and the commit works.

I added a single before-commit script that logged time and args and always returned 0 and it worked too.

ie:

#! /bin/sh

echo "date: $(date)" >> fossil_hook.log
echo "args: $@" >> fossil_hook.log
echo "" >> fossil_hook.log

exit 0

So the question is what's the problem with commit_no_xxx.ss?

When it's triggered in the past, it's always been before any of the round-trip output that i pasted in the original "bug" report. That is, to my uninitiated eye, it looked like it blocked the commit before any syncing etc.

This is entirely a bug with commit_no_xxx.ss, sorry to waste everyone's time.

The problem is that it doesn't properly parse rename edits resulting in running these commands through a shell during commit:

/usr/bin/fossil diff -i -c 0 troppo/pt.sls  ->  troppo/pieces.sls
/usr/bin/fossil diff -i -c 0 troppo/buffer.sls
/usr/bin/fossil diff -i -c 0 tests/pt.ss  ->  tests/pieces.ss

Since the top and bottom are malformed diff commands, fossil writes an error message to stderr leaving nothing going to stdout which is redirected to troppo/pieces.sls and tests/pieces.ss.

Which is why those are empty at commit time.

I started this thread anonymously, so i'm not sure if i can edit its title. It should be marked solved, or better still, deleted entirely to hide my shame. :)

Sorry again.

Actually, the title is all wrong so tacking [Solved] onto it is unfair to fossil.

Maybe something like "Beware before-commit scripts that modify checkouts" and left up as a cautionary tale...

The problem is that it doesn't properly parse ... left up as a cautionary tale...

Be aware (yourself and those reading this in The Future) that we have never guaranteed stable output formats. Though we do make an attempt to avoid blatant breakage, we don't lose all that much sleep over changing output formats. How renames are rendered, for example, was changed sometime in the past year.

Thanks for following up and finding the problem.

Completely unrelated, if i may offer a tip:

echo "date: $(date)" >> fossil_hook.log
echo "args: $@" >> fossil_hook.log
echo "" >> fossil_hook.log

can be more easily done with:

{
  echo ...
  echo ...
  echo ...
} >> fossil_hook.log

or, more cryptically:

exec >> fossil_hook.log
echo ...
echo ...
echo ...
# stdout is redirected until the end of the script