1 ) When testing Fossil I was first surprised that it seems to ignore empty folders (I have checked the forum about this already). I do not think this is a bad decision (although surprising for people like me who know only svn or git), but I have two questions about this:

a) does someone have a list of all directly or indirectly ignored entities? I do not know if mine is complete.

b) is there a way to perfectly reset a chekin without having to (delete it and) make another one from scratch?

2 ) It also seems that "checkins" themselves go into the entities described in 1)a). I have confirmed this by inspecting the source code.

a) Is this by design? or a side effect of the reuse of a check function or anything else? I would like to know more about the context of this decision if possible.

b) Is there a handy way to put checkins inside checkins?

Thank you again.

When testing Fossil I was first surprised that it seems to ignore empty folders (I have checked the forum about this already).

Fossil versions file content. Directories have no content for it to track.

does someone have a list of all directly or indirectly ignored entities? I do not know if mine is complete.

i'm assuming you mean things ignored by the SCM? It doesn't ignore anything by default except (IIRC) dot-files, but it only tracks things it's explicitly told to (so, in that sense, it ignores anything it's not explicitly been told to track). It can be told to ignore certain things matching a list of globs for various purposes. See the extras command and the the ignore-glob setting.

is there a way to perfectly reset a chekin without having to (delete it and) make another one from scratch?

You cannot undo a checkin in fossil. It's there forever (barring a "shun", but that's "the nuclear option" and not intended to be used unless the world is on fire). By design, history is immutable in fossil. If you mean to reset a checkOUT then see the revert command.

It also seems that "checkins" themselves go into the entities described in 1)a). I have confirmed this by inspecting the source code.

i'm not at all clear what you mean by that so won't attempt an answer.

Is there a handy way to put checkins inside checkins?

Fossil does not support nested repositories/checkouts. It is possible to create nested checkOUTs, but they're only structured as nested for purposes of the filesystem: fossil itself associates no connection between them. For example:

$ mkdir foo; cd foo; f open /repos/foo.fossil
$ mkdir bar; cd bar
$ f open /repos/bar.fossil
# ^^^ will error out because fossil does not *like* to nest checkouts, but:
$ f open --nested /repos/bar.fossil
# --nested ONLY disables the check for an opened checkout but does not
# perform or assume any sort of as association between the two checkouts.

Those are, for fossil's purposes, two 100% independent checkouts. That one happens to be nested under another in the filesystem is just how they're laid out in the filesystem and implies no association whatsoever at the SCM level.

You cannot undo a checkin in fossil. It's there forever (barring a "shun", but that's "the nuclear option" and not intended to be used unless the world is on fire).

There's also purge, but it only works on a single repository. If you have autosync enabled, you have to manually repeat the purge on the parent repo. If another user has a clone of that parent and has sync'd the to-be-purged artifact down to their machine since it was sync'd up, you have to manually purge on the other child, too.

At some point, there are just too many places the artifact lives, so purge becomes impractical. Shun has the benefit that you can at least say "this particular repo will forget the artifact and never accept it again", preventing its further propagation, if that's a trusted central repo.

The purge feature was created to support bundles, which are an extension of the private branches feature, so they don't escape in a way to make this sort of mess. Nevertheless, it can be used with any sort of artifact.

It's best done only on leaf artifacts, on a single machine. Beyond that, you risk sawing branches off the Merkle tree while you're sitting on them.

It also seems that "checkins" themselves go into the entities described in 1)a)

Checkins go into the repository DB file, which is normally outside the checkout directory, so yes, Fossil will ignore that data.

It's legal to put the repo file inside the checkout directory, but Fossil is smart enough not to let you add the repo to the set of managed files, so you cannot commit the repo to the repo:

$ mkdir x
$ cd x
$ f init .x
$ f open .x
$ f add .x     # fossil silently ignores you
$ f stat       # no changes seen
$ f ci
nothing has changed; use --allow-empty to override

I believe this intelligence was added in support of the URI form of fossil open:

$ fossil open https://fossil-scm.org

That creates fossil-scm.fossil in the checkout directory, so Fossil needs to be smart enough to not let you add it to the set of managed files and thus create this never-ending recursive commit loop: the commit changes the committed file so it wants a commit, so you commit the commits to the repo, so it wants a commit...

The VSCode Fossil extension works the same way, cloning remote repos into .fsl within the working directory, doubtless because VSCode was built with the assumption that the VCS is Git, so it's trying to reduce differences.

You might find the newly-reworked Fossil Glossary helpful. It's basically a tutorial of the four key concepts in Fossil, which we've been dancing around in this thread.

Excuse me for my vague questions; their real meaning did not seem to get through.

What I meant by "ignoring" is for example, when you mkdir an empty folder and use the "extras" command, it doesn't appear in the list. Maybe the same for addremove etc... There is a function in the source code that checks for files that can be considered for adding etc. (I cannot write its name here now)

And also, what I meant by "perfectly reset a checkin" is "revert local changes or switch to another checkin so that the actual directory contents are equal to the checkin contents (no empty dirs remaining etc)

I would be happy if you could answer again knowing these facts.

I will post again when I can about 2).

What I meant by "ignoring" is for example, when you mkdir an empty folder and use the "extras" command, it doesn't appear in the list.

Because fossil does not track directories. It only tracks files.

There is a function in the source code that checks for files that can be considered for adding etc.

Fossil has to acknowledge that directories exist but it doesn't, and cannot, track their existence persistently. Its persistent SCM metadata format literally does not account for non-files, nor can it without incompatible changes.

And also, what I meant by "perfectly reset a checkin" is "revert local changes or switch to another checkin so that the actual directory contents are equal to the checkin contents (no empty dirs remaining etc)

See the --verily flag for the clean command, including its warning.

Before running that, you possibly want to do either update SOMEVERSION or checkout SOMEVERSION so that empty directories left behind by the switch to SOMEVERSION are cleaned up after switching versions.

Thank you. I have made tests using this option; please check my newest post.

This means that when you do a fossil open --nested inside a checkout (oh by the way, from the start I meant checkout not checkin, I am sorry), fossil will ignore the whole folder just because it is seen as another checkout.

That is as designed. Fossil does not permit checkouts to overlap into the same directories.

This also seems to mean (indirectly?) that the --verily option of fossil clean won't work as expected (?) because it leaves subdirectories containing checkouts intact.

As designed. When fossil sees that there is a nested checkout, it has no choice but to ignore that whole directory tree. Any other behaviour would lead to bugs and painful misinteractions if a checkout manipulated files belonging to a nested checkout.

So I still do not have a way to reset a checkout directory to its pristine state.

If clean is not doing what you want (keeping in mind that clean cannot safely traverse nested checkouts) then the only option available is to delete the checkout and re-check it out.

About 2) : I would like to know if possible what the team (drh,..) thinks about this special treatment of nested checkout directories.

A nested checkout is a "hard boundary" for fossil. Every invocation of fossil only manages a single repository/checkout database, and nested checkouts are separate databases. It is architecturally not possible for fossil to deal with two checkouts at the same time, so the only safe option it has when it sees a nested checkout is to ignore that whole directory tree altogether. Teaching fossil to be able to deal with a collection of grouped checkouts at the same time would require a major rewrite.

Fossil has, insofar as i can find, only a single exception where it will dive into nested checkouts, and in that case will do nothing more than scan for empty directories: see the clean --allckouts flag.

It is architecturally not possible for fossil to deal with two checkouts at the same time

I'm not sure about that.

The lesser feature —which I think would solve a lot of people's problems — would be to implement the equivalent of this in Fossil:

  $ find . -name .fslckout -depth -exec ../doit {} update -v \;

where "doit" is a tiny shell script I can't work out how to write inline as an argument to -exec:

  #!/bin/sh
  cd "$(dirname "$1")" && shift && fossil "$@"

That is, given a .fslckout path beneath the current directory and some number of Fossil command args, repeat that command in each of those directories.

Left as an exercise to the implementor is finding the top of the tree and going there first, so it doesn't matter where you are in the tree: every operation affects all nested checkouts.

Now for the greater version of the feature, which is the version that I think you're suggesting is impossible: allowing a set of repo-modifying operations to be atomic across all affected databases among two or more nested checkouts.

Consider: if today's Fossil aborts an operation, does it not roll back changes to all of ~/.fossil, $CKOUT/.fslckout and to $REPOPATH? Why then can Fossil not attach all databases used by a nested tree of repo checkouts, create a single big transaction, and apply all operations across them, then close the transaction?

Difficult, yes. Tricky to get right, most certainly. But impossible?

Now for the greater version of the feature, which is the version that I think you're suggesting is impossible: allowing a set of repo-modifying operations to be atomic across all affected databases among two or more nested checkouts.

By "impossible" i meant with today's code. We have 3 well-defined/well-known db names, and any sub-checkouts would require adding 2 additional generated names (a checkout is never opened without its accompanying repo db, as the checkout state refers to the repo.blob table). Any number of algorithms would need to be adapted. SQL code which now unambiguously refers to table names would have to be patched to add a schema name (keeping in mind that the schema names would need to be generated dynamically for sub-checkouts). e.g. (SELECT rid from X.blob) would require a different X for each sub-checkout. Messy, messy.

Sure, we can also shell out commands to deal with sub-repos, but that's an icky way to work and smells like duct tape.

Consider: if today's Fossil aborts an operation, does it not roll back changes to all of ~/.fossil, $CKOUT/.fslckout and to $REPOPATH? Why then can Fossil not attach all databases used by a nested tree of repo checkouts, create a single big transaction, and apply all operations across them, then close the transaction?

Supporting multiple checkout/repo db pairs at once would require significant changes and additions. Not impossible in the larger sense of the word, but with today's code, not a chance, nor is it only a weekend's work.

Any number of algorithms would need to be adapted

I was assuming there was a way in SQLite to serially attach a DB into the default schema without committing the transaction. Thinking more about it, I think we need a REATTACH SQL command to let you keep a DB open but shunt it off to another top-level schema so the default namespace can be reused for another DB open.

The thing is, Fossil never needs to update more than one repo at once to make this work. It just has to iterate through the tree of nested repos, queue up the transactions necessary to update each one, and then bammo, close the transaction to update them all.

This means there needs to be a way of saying "I was referring to repo a.fossil and its associated .fslckout, but I am done with that and am now referring to b.fossil and its checkout DB."

As for implementing the lesser version based on subprocesses, I agree that it isn't perfect, but here I think we have a "perfect is the enemy of the good enough" case. Fossil already implements several operations this way.

It seems that you all think that I am thinking about some kind of recursive checkout system, where all I am thinking about is just storing checkout data as raw data.

I understand the point where Fossil ignores empty directories, but directories full of data are not empty and I still can't understand why they should be ignored by the extras command just because there is a '.fslckout' file inside.

By the way, fossil add can add a nested checkout directory without any problems, although this directory is ignored by the addremove and extras command. (why should a directory be addable using add but not addremove?)

By the way, the --verily option of the clean command does not reset tracked changes, so it cannot be used alone to reset a checkout to the pristine state (forgetting about the not-cleanable checkout data subdirectory). I can get close using

fossil co --latest --force

fossil clean --verily

But I still think there should be a "reset directory to identical checkout data" command.

There is also "fossil revert" instead of the checkout command, but I'm wondering if there are any logical differences between them. Maybe I'll check the source code soon.

There is also "fossil revert" instead of the checkout command, but I'm wondering if there are any logical differences between them.

Fossil's checkout cannot pull individual files: it updates the whole repo to a new version. Revert is most often used to reset specific files to the current checkout version, but it can also be used to pull specific versions of individual files.

There's a fundamental difference between:

fossil co VERSION
fossil revert -r VERSION

namely in that the former will update the "which version do we have checked out?" state to VERSION, whereas the latter leaves that state alone. Thus if you do a revert to a version other than the current checkout version, your checkout will show those files as having been modified (compared to the current checked-out version).

It seems that you all think that I am thinking about some kind of recursive checkout system, where all I am thinking about is just storing checkout data as raw data.

Checkout data may be "just raw data" to you, but for fossil it is anything but.

I understand the point where Fossil ignores empty directories, but directories full of data are not empty and I still can't understand why they should be ignored by the extras command just because there is a '.fslckout' file inside.

Fossil does not allow checkout directories to overlap because That Way Lies Madness. It avoids that in order to keep users from shooting themselves in their proverbial feet. Shot-up feet are invariably a support burden for the project's volunteers.

"Oh, but it's just the extras command, which is read-only!" The output of that command is structured such that it can be piped to external commands, e.g.:

$ fossil extras | xargs rm

By crossing into nested checkouts, extras would be at risk of being complicit in data loss and other grief.

In this project, unlike git, we always err on the side of not making it easy to lose data ;).

By the way, fossil add can add a nested checkout directory without any problems

That may well be a bug (in the form of an oversight), not a feature. i'll take a look at that later on. Perhaps it is intended to be permitted on the grounds that it doesn't introduce a risk of data loss.

(why should a directory be addable using add but not addremove?)

The answer to the second part of that question probably lies in the "remove" part of the equation. It could potentially manipulate files out from under a nested checkout, causing downstream confusion when the user runs fossil status in that sub-checkout and sees changes they didn't expect. Just speculating.

But I still think there should be a "reset directory to identical checkout data" command.

That the feature doesn't yet exist indicates that nobody who hacks on fossil has been itched by it enough to feel compelled to put the effort into implementing it and, subsequently, maintaining it forever. (Alternately, maybe someone has and the feature was rejected by the project lead. Dunno.) Maybe someone reading this will consider it an interesting problem and set out to patch it. Until then, the effect of restoring a pristine state is trivial to achieve by nuking the checkout directory and re-opening the repository.

Checkout data may be "just raw data" to you, but for fossil it is anything but.

If I do fossil add on some data, I expect that this data will be treated as raw data and not interpreted as another checkout or any other special data. And this is actually what happens when I do add but not when I do extras or similar commands.

Fossil does not allow checkout directories to overlap because That Way Lies Madness. It avoids that in order to keep users from shooting themselves in their proverbial feet. Shot-up feet are invariably a support burden for the project's volunteers.

This might be a valid reason (but I would be glad to hear drh's opinion about it)

That the feature doesn't yet exist indicates that nobody who hacks on fossil has been itched by it enough to feel compelled to put the effort into implementing it and, subsequently, maintaining it forever.

I think it might be the case too because I'm quite a maniac. Maybe I will take some time to look into this. Thank you for your answers.

It seems that you all think that I am thinking about some kind of recursive checkout system

My idea is more general than checkout. It would make every command that deals with .fslckout recurse through sub-checkouts to apply the same operation there: update, commit, diff, revert, sync… Until fossil gets that feature natively, I offer frec:

#!/bin/sh
if [ "$1" = "doitonce" ]
then
    shift ; fossil "$@"
else
    find "$(dirname "$(fossil sql '.mode tabs' \
        "select file from pragma_database_list where name='localdb'"
    )")" -name .fslckout -depth -execdir $0 doitonce "$@" \;
fi

With this, you can say:

  $ frec clean --verily && frec revert

…to get the behavior you desire.

fossil co --latest --force && fossil clean --verily

I can't recommend use of the --latest option unless you're working without collaborators, or the project has no branches. Otherwise, there's a risk that it could switch you to another branch, simply because it was updated more recently than your current one:

mkdir x
cd x
f init .x
f open .x
touch a
f add a
f ci -m 'empty a'
echo hi > a
f ci --branch hi -m 'a says hi'
f up trunk
echo bye > a
f co --latest --force
cat a

Guess what the last command outputs?

I'd say this instead to avoid that problem:

  $ fossil clean && fossil revert

I rarely bother with --verily, but you can add it if you want. With suitable "glob" settings, the default action of clean usually does what I want.

think there should be a "reset directory to identical checkout data" command

So wrap the pair of frec commands above in another shell script.

Quite impressive I admit; it ought to be really powerful (and beautiful) if implemented correctly.

What I was thinking is just about treating subcheckout data as raw data (like any other data).

I would like to add that the explanation for the verily option

WARNING: Removes everything that is not a managed file or the repository itself.

is wrong (as explained above) so maybe it is a bug (in the docs if not in the code)?

(This was actually outlined above by another user, but the fact is that Fossil actually allows source trees to overlap, and has to because it cannot control the file system)

Maybe it would be nice to alert the user whenever there are two+ ".fslckout" files found up to the root of the filesystem,

When searching for the key to the door, one stops looking as soon as they've found the key. Fossil does the same once it finds the key to a checkout. At that point, there is no use for it to continue searching, nor has widespread experience suggested that it really needs to.

there are no downsides I think,

Except that every time fossil opens any checkout, it has to read an arbitrary number of extra directories for...

and we can prevent the user from shooting himself in the foot.

... that one user who's been bothered by this so far.

If nested repos are not allowed, there should be mecanisms in place to check this invariant.

If they were disallowed, we wouldn't have the open --nested flag. They are allowed but bring caveats and foot-shooting opportunities with them. Caution is in order.

Also, fossil add should prevent adding checkin folders.

AFAIK, it's supposed to do so already. If it's not, that's arguably a bug but also arguably fossil being overly-picky about what the user has told it to do. Software trying to be too clever invariably leads to annoyance at some point when it stops a user from trying to do something they really want to do. In this regard, fossil is more Unixy than Windowsy, the latter being notorious for clipping the wings of would-be power users.

Except that every time fossil opens any checkout, it has to read an arbitrary number of extra directories for

Is checking a few more directories such a burden? It would if RTOS guarantees were needed, but looking up to the root is a common task (the shell does it often) and should have no big timing penalties. Logic is also the same as before, only the stop condition changes.

Not doing this implies that the invariant is not checked.

If they were disallowed, we wouldn't have the open --nested flag.

No invariant then :) The docs should be clearer then. Nested checkouts are allowed if enabled but not encouraged and there's no follow (to me partial follow is no follow) from the Fossil side.

AFAIK, it's supposed to do so already. If it's not, that's arguably a bug

I can't see how this would not be a bug.

Software trying to be too clever invariably leads to annoyance at some point when it stops a user from trying to do something they really want to do.

Bulk adding a foot-shooting folder if not prevented should at least require a command line option (because Fossil should not silently accept something that leads to some NO WARRANTY pattern)

Windows

not my business (in the true sense of the word) ;)

We don't seem to understand each other quite well so I will just answer obvious things until I (maybe) make some proposal.

nobody but you

The complaint it generates would serve to prompt users of nested checkouts to come to the forum and report, "fossil is saying something scary - am I doing something wrong?"

A bug is a bug if a bug, you can't just ignore things because someone came to say it first. (except that in this case it doesn't seem to be a bug so I will stop discussing this matter)

but software trying to be too clever

ignoring directories

outsmart

I have never said that Fossil should ignore data. fossil add adds silently nested checkouts even if nested checkouts are 1) forbidden according to the docs and 2) unsupported/dangerous.

I just thought that it should maybe give some warning or anything else.

No warranty is voided, but your expectations as a user are not being met

If doing a "fossil add nested_checkout" causes problems and needs precautions, I would like to know about it beforehand.

The solution there is to be selective

Here "bulk" didn't mean "add ." or anything like. Sorry.

A bug is a bug if a bug, you can't just ignore things because someone came to say it first. (except that in this case it doesn't seem to be a bug so I will stop discussing this matter)

A bug is something which result in an incorrect result. Fossil doing exactly what it's told to do is not a bug, even if it's defying user expectation.

If doing a "fossil add nested_checkout" causes problems and needs precautions, I would like to know about it beforehand.

Trying to open a nested repo causes fossil to insist that the user pass in the --nested flag. That in itself is a warning that "something is different." What we're missing is what differs, but see below...

Nested checkouts are not "unsupported," but they do come with caveats and it's expected that people using them are ready to deal with those caveats. If you can suggest where such caveats should be documented in detail, we can get them added to some docs (new docs, if needed). Few of us actually use nested checkout, or we use them in ways which don't cause us to trip over them like you have. Therefore my suggestion would be that you, as the one whose most recently been bitten by them, write up some docs about the various issues/caveats, and we can add them to the site and link to it from warning open emits when it insists on requiring the --nested flag. Someone who hasn't tripped over them (namely me) is not in a great position to write up accurate docs about the caveats.