The key innovation of the venerable CVS version control system was the ability for two or more developers to make changes to the same file at the same file, then have the version control system merge those changes together later. Prior to CVS, you would have to "check-out for editing" a file, then make your changes, then commit. And while you had the file checked out for editing, nobody else was allowed to modify it.

People are fond of throwing shade on CVS these days, and it does have its limitations, but those of us who had to use the systems that came before will always remember it at a huge innovation. The ability for two or more developers to work on the same file at the same time was a huge improvement, and for whatever its weaknesses, we will always remember CVS fondly for introducing the concept.

But the concurrency feature of CVS only works for files that can be merged, which normally means text files. That works great for source code. But what if your repository contains binary files like images, videos, spreadsheets, presentation decks, etc. that cannot be merged?

For binary files, it would still be convenient to be able to "lock" a file for editing, so that two developers do not accidentally work on the same file at the same time. I'm wondering if this is a feature that should be added to Fossil.

This is inspired by an actual need. I have a (private) repository of LibreOffice presentations that I am co-editing with Dan Kennedy for an upcoming business meeting, and we need a better solution for coordinating edits to the various *.odp files.

Note: Eric Sink's (now defunct) version control system "Veracity" was designed to work for video-game development, which contains lots of binary resources, so it had a lock-for-edit feature. This is not a new concept.

So if something like this were to be added to Fossil, how would it work? Clearly it would require a server. There has to be a central lock authority and so lock-for-edit would not work with disconnected operation.

Assuming an appropriate server, perhaps something like this:

1. Designate some subset of files as "unmergeable", perhaps using a list of GLOB patterns. "Unmergeable" files are checked out read-only.

2. When you want to edit an unmergeable file, you run "fossil lock FILENAME" to acquire an edit lock from the server. If successful, this changes the permission bits on the file to read/write

3. The first "fossil commit" of the file, drops the edit lock and changes the file permissions back to read only. Or you can also run the "fossil unlock" command.

Issues:

1. How to deal with the same file that diverges in separate branches?

2. What if one developer locks a file then goes on vacation? (This was a common problem in the pre-CVS days.) There needs to be some way to break a lock.

This topic came up a few years back on the mailing list, but i don't recall a consensus on how it might work in a DVCS.

What if one developer locks a file then goes on vacation?

What if... (i'm just spitballing here)...

What if the locks are stored in .fossil-settings/locks, 1 line of 2 records per file:

dan presentation.odp
drh otherfile.ods

(Alternately, instead of a flat file, use a dedicated sqlite mini-db.)

When doing an update/checkout, for each filename in locks, if the user name does not match the current fossil user, then make it read-only and emit a note to the user. For each file in locks where a username matches, or which has no user name (maybe a placeholder of * for the user name in that case?), make the file read/write.

To unlock a file, locks would need to be edited and committed.

It wouldn't prevent a user who edited a file before it was added to locks from editing it, however:

1. Dan starts editing foo.odp.
2. You add foo.odp to locks and check that in. Fossil sees your name next to foo.odb and (if needed) marks you copy read/write.
3. You edit foo.odp.
4. Both of you try to commit.
5. Chaos ensues.

Maybe locks could be unversioned configuration-level stuff, rather than real repo artifacts (but then synching it may become an issue?)? Having a repo artifact for locks seems a bit painful but it would raise the visibility via the timeline commit entries for the lock file.

Locking was a CVS feature which tended, in my experience, to cause more headaches than it solved, but there is inarguably a use for it.

(Alternately, instead of a flat file, use a dedicated sqlite mini-db.)

Which would obviously be a bad idea because that file would be binary.

I have a (private) repository of LibreOffice presentations that I am co-editing with Dan Kennedy for an upcoming business meeting, and we need a better solution for coordinating edits to the various *.odp files.

I covered that in my article Image Format vs Fossil Repo Size: automate the unpacking and reconstitution of that Zip file — which is what a *.odp file is — and check in only the unpacked form. Now you've got text information that Fossil will be more likely to merge correctly.

This will work under the same rules as with C: if those with checkin rights on that repo are working on different slides, the XML data file being updated should change in different enough places that Fossil's merge algorithm won't have any problem doing the merge.

This idea extends to the graphical assets you embed into the presentation: use uncompressed BMP or TIFF for bitmapped graphics, and uncompressed SVGs for vector graphics. If both of you update such a graphic and re-embed it into the presentation, the changes should merge as long as you're editing sufficiently well-separated parts of the graphic file.

There has to be a central lock authority and so lock-for-edit would not work with disconnected operation.

It should suffice simply to make it dependent on having autosync enabled.

Here's a fun wrinkle: what if you're using Fossil in a truly distributed fashion? If I clone from www2.fossil-scm.org and you clone from www3.fossil-scm.org, then we both edit the same unmergeable file and check our changes in within the window where these two servers sync, each server will allow the checkin as uncontested, and it will be up to the two servers to work out the actual merge. If they fail, then what?

Does the lock need to propagate through the whole clone clone tree, minus the leaves, else you don't acquire the lock?

Now you've got a distributed consensus problem. What if www.fossil-scm.org is partitioned from www2 and www3 at the moment that a third editor who cloned from that repo is now attempting the lock? Do we then need Paxos or Raft logic to decide that two servers is enough to acquire the lock as long as you're doing it from within the www2 + www3 quorum? Does the one who cloned from www fail to acquire locks until that server rejoins the quorum?

The first "fossil commit" of the file, drops the edit lock and changes the file permissions back to read only. Or you can also run the "fossil unlock" command.

I often check in multiple changes to a file in a single work session. It would be quite annoying to be forced to reacquire the edit lock multiple times during that work session, once per checkin.

The read-only file attribute won't stop most programs from allowing you to edit the file, just from saving it. And not even always then, since it is the permission of the directory that matters in replacing a file outright. That is to say, some programs update a file by writing to a temporary, deleting the old one, and renaming the temporary. Read-only file attributes won't help you there.

To cast away the "concurrency" in the CVS initialism, it would be better if an unmergeable file is not merely marked read-only, it should not be present in the checkout until the lock is acquired. On acquiring the lock, the unmergeable file appears, and now it's yours to modify, exclusively. On unlock, the file disappears, provided there are no uncommitted changes.

This only works for files that have no dependents. For instance, a web app with PNG images for inline graphics. If the PNGs are marked unmergeable, the web app doesn't run properly because they can't even be read by the web server until the lock is acquired.

There's another new problem: what if someone has an unmergeable file open in the editor for that file (LibreOffice in our example here) but hasn't acquired the lock. Maybe they had the lock shortly before, checked in their changes, and then had their lock automatically taken away, as you suggest. Now they save the file, which creates a difference between the edited version and what's checked into the repo. On acquiring the lock, Fossil will try to merge the local edits with the remote edits. Haven't you just bought the original problem all over again?

Which brings me back to the unpack-and-reconstitute solution. Not only does it avoid this problem of unmergeable files, it avoids ballooning the repository size on each checkin. See the pretty graph in the middle of the document.

Threads are evil because locking is hard. Distributed locking is even harder. It is worth jumping through hoops like the one I show in that article in order to avoid this pain.

Locks are a good idea. I posted on this topic on the old fossil email list a long time ago. I think locks can be very well implemented on fossil if instead of thinking of them as locks think of them as a semaphore. The goal should be to synchronize access to minimize or eliminate collisions, not to absolutely prevent edits. Rock solid, unassailable locking is fully at odds with a distributed system but I argue you do not need solid locking - cooperative editing would be so much easier where you don't have to do a bunch of email handshaking to start edits on a non-mergeable file.

Files would be set to a lock-controlled state:

fossil lock-control schematics/*.sch

A packet recording the lock-controlled state would be added to the packet store, the files would be set to read-only and a sync would be triggered.

To edit the file you do an edit-reserve:

fossil edit-reserve src/engine-controller.sch 5d

This would sync, check that the file is not already edit-reserved, create an edit-reserved packet, sync again and set the file to writable.

On commit the edit-reserve is cleared (with a clear message to the user).

This would be very very useful and yet would not violate the distributed nature of fossil. Yes, there would be corner cases where you might collide on getting a lock but this is no more onerous than getting a fork.

Just my $0.02

I posted on this topic on the old fossil email list a long time ago.

Can you dig up the thread, please? There might be nuggets we can mine from it.

instead of thinking of them as locks think of them as a semaphore.

You're going to have to be clearer about the distinction you're trying to make. Locks are often implemented using semaphores.

Rock solid, unassailable locking is fully at odds with a distributed system

I wouldn't say that. What is true is the CAP theorem. It's an immovable object here, and we only get to choose how we want to deal with it. We can't bypass it.

I think we want a CP solution here: every reader (i.e. fossil up) must always get the current state of the repository in the face of network partitions, as long as they're connected to a quorum of nodes.

To extend my example above, if you've cloned from www.fossil-scm.org and it's partitioned from www2 and www3, you simply can't update your repo until it rejoins the quorum if you're using this new feature. That is to say, I think we should sacrifice availability to get consistency and partition tolerance.

Fossil today is an AP system: on fossil update, you won't necessarily get the current version of the repository if there's a network partition or if the network is unavailable from your location.

We have to have C in this instance to prevent merge conflicts. Our only choice, then, is whether we take A or P in our "pick 2" choice.

That means we have just one alternative, a CA system, which means it's intolerant to network partitions. In the 3-server system behind fossil-scm.org we'd wequire that all servers be available before any update is allowed to happen.

cooperative editing would be so much easier where you don't have to do a bunch of email handshaking to start edits on a non-mergeable file.

We have that today, and it is indeed excellent. The trick is, it only works in practice when automatic merging is possible.

A packet recording the lock-controlled state would be added to the packet store, the files would be set to read-only and a sync would be triggered.

This vague discussion of "packets" doesn't explain how you're preventing merge conflicts. All I'm getting from this scheme could be restated as "there should be locks, and they should work." Okay, but how?

you might collide on getting a lock but this is no more onerous than getting a fork.

Are you suggesting that Fossil should just keep both versions of the unmergeable file? That's fine from the standpoint of saving all of the user's work durably, but I don't see that it actually solves the key problem here, which is that changing a single character in a single LibreOffice Impress slide is likely to change thousands of bytes elsewhere in the file, almost guaranteeing a merge conflict, even when those working on that file are working in very different parts of the file.

fossil lock-control schematics/*.sch

I'm curious which EDA application you're using there. Several of those that use *.sch as an extension for electronic schematics — EAGLE, KiCad, and gEDA (gschem) included — use text-based file formats in current versions, so they don't have the core problem leading to this thread's topic.

Old versions of EAGLE used a proprietary binary file format, which is one of the main reasons I upgraded: so I could store them in a VCS with minimal repo size bloat.

it would be better if an unmergeable file is not merely marked read-only, it should not be present in the checkout until the lock is acquired

On further reflection, I think you want both behaviors: a read-only copy that Fossil blindly creates or overwrites from the repo and ignores changes to, plus a lockable-and-writeable version elsewhere in the checkout tree.

For the sake of argument, let's say the scheme is a -read and -edit pendant on each file marked as unmergeable:

  $ echo '*.odp' >> .fossil-settings/unmergeable-glob
  $ fossil add foo.odp .fossil-settings/unmergeable-glob
  $ fossil ci -m initial
  New_Version: 648018cd0aed8...
  RENAME foo.odp foo-edit.odp      # lock implicitly acquired
  UPDATE foo-read.odp
  $ soffice foo-edit.odp
  $ fossil ci -m "hacked on the prezzy"
  New_Version: 1feadd75f0fbd6bae2...
  UPDATE foo-read.odp
  $ fossil unlock foo-edit.odp
  DELETE foo-edit.odp

I think you want another Fossil command for overwriting the read-only versions from the editable versions without doing a checkin. Something like fossil update --locked.

This scheme would solve my web app problem: the web app would be written to reference the read-only copy of each checked-in unmergeable file, and the app developer would work on the editable copy while the file is locked. When the file is unlocked, the editable version disappears, leaving only the read-only copy behind, which should never be opened in an editor.

If you edit the read-only copy anyway, Fossil can see that the mtime or file size is different than expected, so it can refuse to update the checkout tree, warning the user that doing so would overwrite unversioned changes. Basically, do what Fossil already does when you "open" into a tree that holds a closed checkout.

The unmergeable-glob setting should extend binary-glob: if a file matches a pattern in the former, it should behave as if you'd added it to the latter as well. That is, don't gripe about *.odp being binary data. That's implicit in marking a file as unmergeable.

changing a single character in a single LibreOffice Impress slide is likely to change thousands of bytes elsewhere in the file, almost guaranteeing a merge conflict

I just tried the experiment: changing one letter of a LibreOffice presentation slide in a file that's 10 kB in size caused an 8 kB increase in the size of the Fossil repo. That means 4/5 of the bytes in the file are changing on each single-byte change in the input!

Can you dig up the thread, please? There might be nuggets we can mine from it.

That would be...

I expected just a URL, but that was under the assumption that there were replies to it. That is not the case here, alas.

Removing write permissions to the locked files is adequate control.

How do you answer my objection above, then? (i.e. "Save" = write to temp, remove original, rename temp.)

The wiki page holds a sexp with lists of lockable files, lock state and if locked who owns the lock.

Why put S-expressions in a wiki article, when we have a perfectly good relational DBMS?

Even if that's a good idea by some standard, what keeps two Fossil instances from updating two nearby S-expressions, thereby causing a merge conflict when the wiki article syncs?

Data storage issues aside, this solution appears to put all of the logic on the leaf host that wants edit permission. That's not a lock, that's a declaration of intent. "I'm gonna modify this file now, please no one else touch it, kthxbye." Nothing stops another host from making the same declaration. Even if fossil lock does an autosync before allowing the lock, it still creates a race condition.

Thanks to Kees Nuyt for finding and reposting my previous message on this subject.

My responses to the various points made:

1. By semaphore I did not mean the computer specific meaning. I mean it simply as a communication from one developer to others that they are working on a file, please no one else touch it until I'm done.

2. Locking in the sense of a tightly controlled system such as Design Sync, RCS, etc. is simply not possible in a distributed way. My point is that a less stringent locking system run on top of a sync mechanism such as what Fossil provides would still be very very useful.

3. This discussion is mixing two distinct problems. I believe it is helpful to separate them. Textual mergeablity is not the only factor when it comes to usefulness of locks. There are text based files where merging causes more trouble than it is worth. The discussion of enhancing file storage and improving automatic merging (e.g. dissecting Libre Office files for better diffs) is important but should, IMHO, be kept separate from the merit and implementation of a locking system.

4. My use of the word "packets" was unfortunate. I meant "artifacts". I stole the artifacts idea from fossil, wrote a small library which I called pkts and forgot the original name. BTW, thanks Richard for artifacts!

5. My post muddied the water with an inadequate description of my views on how locking could be implemented in fossil. At the highest level think of how, using fossil and it's artifacts and sync system, you can have users transparently communicate that they are working on a file and for the moment no one else should touch that file. I'll try writing a clearer description and post back here some time in the next few days.

6. I have used both EAGLE and gEDA in miscellaneous projects and this is a perfect case of a textually mergable file that I almost certainly do not want automatically merged.

7. I wrote refdb (http://www.kiatoa.com/fossils/refdb) to make it easy to store gnumeric spreadsheets in a SCM friendly way. It works pretty well. I'm pointing it out as an example of one way of solving the Libre Office problem - convert to a plain text format and source control that. Maybe a similar tool could be written for Libre Office - convert to and from asciidoc for example. By the way, with a refdb controlled by fossil it is really easy to see who changed a value in a cell and when they changed it, try that with the binary files.

Why put S-expressions in a wiki article, when we have a perfectly good relational DBMS?

The relational DBMS is local to your machine and for locks to be useful they need to work across the network.

I was attempting to implement a locking system using the fossil wiki as a sync friendly mechanism for sharing the state of the file locks. Obviously a native implementation using artifacts directly would be far more efficient.

Data storage issues aside, this solution appears to put all of the logic on the leaf host that wants edit permission. That's not a lock, that's a declaration of intent. "I'm gonna modify this file now, please no one else touch it, kthxbye." Nothing stops another host from making the same declaration. Even if fossil lock does an autosync before allowing the lock, it still creates a race condition.

Yes, this approach would not meet the requirements of stringent locking but I think it would be sufficient for many situations and I think any stringent system will be onerous to install and maintain due to the need for a central server.

Locking in the sense of a tightly controlled system such as Design Sync, RCS, etc. is simply not possible in a distributed way.

That's not true. You just have to live within the limits of the CAP theorem.

This is very well-mapped territory these days, with so many companies running databases distributed across Internet data centers. We know how to do this correctly now, with mathematical rigor.

This discussion is mixing two distinct problems.

I agree: merge-ability and locking are fully orthogonal.

Not only are there text files you might want to mark "unmergeable" so they require a lock to edit, you might have binary files for which none of the workarounds in my article apply.

I have used both EAGLE and gEDA in miscellaneous projects and this is a perfect case of a textually mergable file that I almost certainly do not want automatically merged.

I've never actually tried it, since these are all solo projects.

If the EDA tool doesn't do the right thing with a Fossil-merged file, I'd consider it grounds for filing a bug. It should work. If you add a resistor to the input of this op-amp over here and I change the value of this cap over there, and we check our changes in from different machines, the result should still be a legal and correct schematic.

Maybe a similar tool could be written for Libre Office - convert to and from asciidoc for example.

ODF is already text-based. The core XML file just happens to be zipped-up with a bunch of other files of various other formats, some of which change with each save and some which stay the same across multiple saves.

Note that LibreOffice supports *.fodp presentations which are flat XML files (so, just text) and may be automatically merge-able in some cases. (The GNU Parallel book source was published as flat XML in VCS probably for similar reasons of allowing sensible diffs.)

Unfortunately, I have never tried this workflow with merges so I cannot promise that automatically merged *.fodp files are going to be sensible for LibreOffice.

Thanks for the info on *.fodp files. I didn't know they existed. Interesting.

Nevertheless, I don't think they will help. (1) The file contains some very long lines (thousands of characters) and so might not merge well. And (2) even if we were able to merge parallel edits, what do you think would be the chance the the result would be valid XML that LibreOffice would understand? I haven't actually tried it, but my guess is that a successful merge would be unlikely. And the XML is not understandable by humans (or, at least not by me) so if a merge conflict did arise there would be little hope of repairing it.

Hence, even with *.fodp files, I think there is still a case for having the ability to lock a file for editing.

That's not true. You just have to live within the limits of the CAP theorem.

This is very well-mapped territory these days, with so many companies running databases distributed across Internet data centers. We know how to do this correctly now, with mathematical rigor.

But the point is that we are NOT within the limits of CAP theorem. We have users in a 12 hour away time zone with occasionally unreliable network and overloaded machines and other real-world problems. If two people attempt to get a lock within a few seconds of each other using the distributed approach to locking then BOTH users will have a lock and will potentially proceed to work away completely unaware that they are working in parallel. For context, on one of our busiest fossil repos (20-30 commits/day), we get a fork every month or so due to the intrinsic fossil commit race.

However I still argue that this limitation is no reason for not implementing a locking mechanism (better called a in-use hint maybe?) in fossil. In fact having the in-use hint would be super useful for us - even if the in-use hint would be race prone at times. Similar to forks Fossil can put out a noisy message when there is a in-use hint collision so the user can address it.

ODF is already text-based. The core XML file just happens to be zipped-up with a bunch of other files of various other formats, some of which change with each save and some which stay the same across multiple saves.

Machine generated XML is not pragmatic for parallel editing in many cases:

1. It is not very human readable. Contrast an XML file with asciidoc or markdown to understand my point. The noise level is so high in XML that any non-trivial merge conflict is going to be very painful to resolve.

2. Some XML based tools don't bother to preserve the order or the records when output to a file.

we are NOT within the limits of CAP theorem.

The CAP theorem is a mathematical proof about a fundamental limitation of computer science; it is not a boundary you can get inside. You can choose to push your application towards the limitation or stay away from it, but you cannot be "outside the limit."

The CAP theorem is to computer science as the speed of light is to physics: all things in each system are inside or at the limit.

I don't believe I'm being pointlessly pedantic. If you don't think about this sort of thing correctly, you come to incorrect conclusions.

We have users in a 12 hour away time zone with occasionally unreliable network and overloaded machines and other real-world problems.

That's exactly the sort of problem that distributed consensus algorithms were created to deal with.

Reflect on the fact that Paxos was published in 1989. You think your network is slow and unreliable today? In 1989, UUCP was still being used daily to move information around the globe!

If two people attempt to get a lock within a few seconds of each other using the distributed approach to locking then BOTH users will have a lock

Not if you're using a distributed consensus algorithm like Paxos or Raft.

The real downside here is that such algorithms require a quorum of participants, which can be no smaller than two-of-three, which entirely rules out fossil set autosync 0. That is just a restatement of the CA vs CP vs AP observations above.

the intrinsic fossil commit race.

There is no commit race if you're using the central-and-clones model and you have autosync enabled. You must have deviated from that in one or both of those ways:

• If you've got developers that run with autosync off, either by preference or because they really are offline at some point in the working day and still need to commit, have them do such work on branches particular to each developer. Then there can be no fork. When they get back online, they can merge their work into the common working branch.

• If instead the problem is that your cloning hierarchy is more than two levels deep, ask whether you really need to allow that. Fossil's prevention against forks is currently a two-party conversation: if the central repo refuses the commit, the leaf will roll back its commit as well. That doesn't work when one of the central repo's clones is itself cloned and then checkins are made against that third-level repo. It only asks whether the second-level repo allows the commit; the central repo isn't involved in the decision.

That gets us back to the Paxos/Raft/CAP discussion. Consider this three-level cloning scheme:

   HQ → remote → dev1
               → dev2
               → etc.

That is to say, the primary repository is hosted at headquarters, then there is a remote development office that has a local clone of that repo, and then each developer at that office makes their clone and works from that. Today's Fossil does risk creating forks with such a scheme because it's an AP system: it's highly-available (A), and it's partition-tolerant (P), but all parties don't always see the same data. We've chosen to trade away consistency (C).

I don't think anyone wants Fossil's locking scheme to be CA system. That would mean every clone in the chain would have to be online and communicating before a lock is allowed to be taken or released. That's what it means for a system to be partition-intolerant.

That leaves us the CP alternative: consistent (C) and partition-tolerant (P), but not always available (A). In the context of locking, this means that everyone agrees that a file is locked or unlocked; no one is ever in doubt. In other words, a lock cannot be taken or released until all parties involved are in communication. When one or more parties cannot communicate, everyone can still work with the state last agreed upon when they were all in communication.

For locking, I think you only have to consider the leaf clone that actually holds the lock and all ancestor repos in the cloning chain up from there. For example, if dev2 in our diagram above takes a lock on file foo, he doesn't have to ask dev1 if he can have the lock, only remote and HQ. Once dev2 has the lock, dev1 and the rest at that office learn about it the next time they sync with remote. There is no race, because they are bound by the same restrictions: they have to ask remote before they can take a lock on foo, and that can only happen if HQ also agrees.

Contrast an XML file with asciidoc or markdown to understand my point

It turns out that there are Markdown-based presentation tools. I would also support switching to one of those over continuing to fight with the problems that LibreOffice presents.

That said, this is still a worthwhile discussion because we aren't always given such an alternative. I can't see there ever being anything like Markdown for electronic schematics, for example. When in such situations, we don't get to argue about the esthetics of XML vs something else; we only get to choose how to accommodate such files in our Fossil repos.

Some XML based tools don't bother to preserve the order or the records when output to a file.

Unless I am wrong in my sense that such cases amount to a tiny minority of XML-based file creators, I don't see a good reason to base any of our decisions here on the existence of that tiny minority.

a lock cannot be taken or released until all parties involved are in communication.

Sorry, that's wrong. I should have said that for a CP locking system, a cloning tree as diagrammed above, and a Raft/Paxos type distributed consensus algorithm, a lock can be taken or released only when a quorum of nodes are in communication.

This means that if there is a partition between HQ and the remote development office, no one using the HQ repo directly can make any changes to the locking status of files until at least the remote office comes back online. (Again, two-of-three.) We're sacrificing availability (A).

Likewise, if a remote developer is offline — that is, she is partitioned from the remote office's Fossil server — she cannot change the locking status on a file in a CP locking system.

Where it gets really tricky is when we extend the diagram above:

   Chicago → Hyderabad → dev1
                       → dev2
                       → etc.
                Sydney → dev9

Now we've got two remote offices, which means the minimum quorum size with Paxos and Raft is three-of-four. Let's insert partitions to consider the consequences:

1. If any developer is offline, they cannot change the locking status on a file. They have to go with the info from the last time they were in communication with at least their local office's server and HQ.

2. If HQ (Chicago) goes offline, no one can change the locking status on any files in the system as-drawn, if this is to be a CP system, because no one can form a quorum.

3. If there is a way for Sydney and Hyderabad to communicate directly, they could arrange to sync their repos, fixing the problem, because now their developers can from 3-of-four quorums in making locking decisions. Chicago is left to figure out what happened after they come back online. (Distributed systems often do such syncing rearrangements automatically.)

4. Another way around the problem is to have a second central data center. Call it New York. The minimum quorum size is still three, but now it's of-five. With that extra redundancy at the highest level, either Chicago or New York can go offline without interruption of service to any developer that's still online with respect to their local development office's Fossil server.

This gives us a solution to the "lock taker goes on vacation" problem: Chicago, Sydney, and Hyderabad can get together and decide the file is no longer locked. When the developer comes back from vacation and syncs, they'll be told their lock has been taken away by consensus among the quorum left behind. It doesn't matter where that developer is: in a CP Raft system, every developer in the network must form a quorum with at least two of the nodes who came to that consensus.

This solution only works in practice if the capability to break a lock is abnormal, either by policy or by server enforcement.

For the enforcement case, the Fossil capability to make/release locks should be separate from the capability to break another's lock. Only special users might get the capability to break locks.

Sites with high levels of mutual trust can give this capability to every developer, along with advice not to go taking away others' locks without due reflection.

I think this command scheme makes sense:

   $ fossil lock foo
   $ fossil unlock foo
   $ fossil lock --break foo
   $ fossil unlock --break foo

The first two commands are those used almost all the time in practice.

The third command forcibly re-locks a file currently locked by another user. "I need to change this file, and the current lock holder has disappeared and so cannot release it, so I'm taking their lock away and replacing it with my own."

The fourth forcibly breaks a lock held by another user. "I don't need to change this file myself, but the current lock holder can't be allowed to keep holding onto it."

All of these commands will fail if there is no quorum of repos available. A lone dev sitting on a beach in Fiji without Internet access can't make, release, or break locks on files.

I think this command scheme makes sense:

Now I'm thinking about quorum management. New commands:

   On server fossil.chicago.example.org:
   $ fossil init repo.fossil
   $ fossil server fossil.repo &

   On developer box dev0.chicago.example.org:
   $ fossil clone https://fossil ../repo.fossil
   $ fossil open ../repo.fossil
   $ fossil lock foo                       # 1

   On server fossil.hyderabad.example.org:
   $ fossil clone https://...chicago...
   $ fossil serve repo.fossil &
   $ fossil quorum                         # 2
   $ fossil quorum add                     # 3
   fossil.hyderabad.example.org

   On developer box dev1.hyderabad....
   $ fossil clone and open and cd and stuff
   $ fossil lock bar                       # 4

   Back on server fossil.chicago.example.org:
   $ fossil quorum add -R repo.fossil
   fossil.hyderabad.example.org
   fossil.chicago.example.org

   Back on developer box dev0.chicago...
   $ fossil unlock foo                     # 5

   On server fossil.sydney.example.org:
   $ fossil clone https://...chicago...
   $ fossil serve repo.fossil &
   $ fossil quorum add                     # 6
   fossil.hyderabad.example.org
   fossil.chicago.example.org
   fossil.sydney.example.org

   On developer box dev9.sydney.example.org:
   $ fossil clone and open and cd and stuff
   $ fossil lock bar
   DENIED  Lock held by user1
   $ fossil lock --break bar               # 7

   Back on server fossil.chicago.example.org:
   $ killall fossil

   And then on dev9.sydney.example.org:
   $ fossil ci bar -m 'file bar updated'   # 8
   $ fossil unlock bar
   DENIED   Quorum not achieved
   $ fossil unlock bar --verbose
   DENIED   Quorum not achieved
   ONLINE   fossil.sydney.example.org
   OFFLINE  fossil.chicago.example.org
   OFFLINE  fossil.hyderabad.example.org   # 9
   $ fossil unlock --break bar
   DENIED   Quorum not achieved

   New server fossil.newyork.example.org:
   $ fossil clone https://...sydney...
   $ fossil serve repo.fossil &
   $ fossil quorum add
   fossil.hyderabad.example.org
   fossil.chicago.example.org
   fossil.sydney.example.org
   fossil.newyork.example.org
   $ fossil quorum check
   OFFLINE  fossil.hyderabad.example.org
   OFFLINE  fossil.chicago.example.org
   ONLINE   fossil.sydney.example.org
   ONLINE   fossil.newyork.example.org

   Back on dev9.sydney.example.org:
   $ fossil sync
   $ fossil unlock bar                     # 10

Footnotes:

1. The lock by user0@dev0 is allowed because no quorum has yet been established, and no one else has a lock on file foo.

2. No results from this command because the repo it cloned from included no quorum info with the repo data.

3. The Hyderabad remote development office adds itself as the first member of the consensus quorum. If autosync is enabled, Chicago learns of this change immediately, if it's still visible from Hyderabad. Command reports that there is only one quorum member, itself. (It implicitly runs fossil quorum for you at the end.) Note that Chicago doesn't become part of the consensus quorum until further down, after point 4.

4. Lock is allowed because no one else has bar locked, and the maximum quorum size is 1, which is treated the same as the "no quorum" case: in both, a clone only has to get permission from its parent to acquire or release a lock.

5. Unlock is permitted with quorum size = 2 because that must still be treated the same as the no-quorum and quorum size = 1 cases, because no majority can form with only a maximum of 2 quorum members. In other words, quorum sizes less than 3 are advisory locking. Yet another way to say it is that Fossil locking is AP at such quorum sizes.

6. For the first time, we have the potential to form proper consensus quorums. CP rules take over from Fossil's normal AP mode of operation, for locks at least.

7. user9@dev9.syndey.example.org called the Hyderabad office and learned that user0@dev0 got hit by a commuter train and won't be back in the office for 4 weeks while he's going through PT, so she's decided to break and re-establish the lock under her user's control so work can proceed.

8. Checkin allowed, even though Chicaco is now offline, because Fossil checkins remain AP. (Maybe. We can argue this separately.)

9. Unlock not allowed because no consensus can form with Chicago offline, so we asked for full details by adding --verbose. It tells us that Chicago is offline, which explains the problem: we need 3 nodes, including our own leaf clone, in order to make this change to the file's lock status, and we've only got 2. Hyderabad is considered "OFFLINE" because we haven't given Sydney a way to see Hyderabad directly. At this point, every lock change has to involve Chicago. This is why our attempt to forcibly break the lock also fails.

10. Now the unlock succeeds because with the sync we learned of the New York server, which means we can now make three-of-five quorums to back lock change decisions. Note that Hyderabad is also considered offline with respect to New York and Sydney: its server is behind a firewall, not able to serve out as those at Chicago, New York, and Sydney can.

    Maybe Sydney is normally that way, too, but a hole was punched in its firewall only long enough to get the initial clone through, then Sydney's server was hidden away again and re-pointed at New York as its clone parent. Fossil supports this already, today.

    Regardless, work can now proceed everywhere but at Chicago itself, until its network connection comes back up, allowing the clone on dev0 to form consensus quorums. user0 on that machine can still make regular Fossil checkins if we decide that stays AP under this scheme, but he can't make changes to locks because it was known at Chicago that we have a CP quorum before we lost our Internet connection.

    This does imply that a site can also "fix" this by removing members from the quorum list until it drops to 2 or smaller, causing Fossil to fall back to advisory locking. (AP mode.)

A branch and a fork are the same thing. Fossil has historically considered the difference between the two terms to be "one of intent": "branch" is used when the fork is intended and "fork" is used when it's not.

So, yes, other SCMs do fork, they just don't call it that.

So, yes, other SCMs do fork, they just don't call it that.

Maybe it would be better if Fossil used "unintended branch" instead. This would also avoid confusion with other uses of "fork", like "InkScape is a fork of Sodapodi".

I'm sure that for most potential users, the message:

Would create an unintended branch

is clearer than:

Would fork

No specific terminology is ever going to be intuitive to all people, but fossil has used its current terms for more than a decade without them causing widespread confusion. As a brand new user, "unintentional branch" would be no clearer to me than "fork", and both cases would require me to go look up/ask about the meaning. Changing the error message would like cause most long-time users to scratch their heads at least once upon encountering it, plus it would require reworking other documentation. Also it's a mouthful: six syllables, compared to fork's one.

In a distributed system, like Fossil or Git, there will arise times when two or more people, perhaps working remotely or disconnected, will commit against the same prior check-in and thus product a fork. Fossil allows you to see all of the forks that have happened:

https://fossil-scm.org/fossil/timeline?n=all&forks

Both Git and Fossil are distributed, and both use a directed acyclic graph (DAG) as their underlying data structure. And so both can fork. The difference is Fossil stores content in a relational database, and so forks in Fossil are very easy to deal with, whereas Git uses a bespoke key/value storage engine that lacks any kind of query language, and thus has difficulty dealing with forks.

Forks in Git lead to the dreaded "disconnected head state". This is a bad place to be and is likely the inspiration behind the famous XKCD cartoon about git. Because Git does not handle forks well, Git culture has evolved to emphasize avoiding forks. The tooling and best practices of Git work to help you avoid forks, and the Git community has picked up the idea that forks are inherently bad and evil.

Fossil uses a relation database for storage, and thus has no "disconnected head state" problem. Forks in Fossil are trivial to deal with. Consequently, no culture devoted to the avoidance of forks has grown up around Fossil in the way it has with Git. It just hasn't been needed. Forks happen. They are quickly resolved (or perhaps promoted into a separate branch). And we move on. No stress.

With that in mind, it is easy to see how somebody who is coming from a Git culture where they have been thoroughly inculcated with the idea that forks are evil, might visit Fossil-land and see our lackadaisical attitude toward forks and recoil in horror. This does not mean that Fossil as a "fork problem". The problem is that the user has been mislead to believe that forks are fundamentally difficult. The problem is that the user believes that something that Git handles poorly must necessarily be handled poorly by all other version control systems too.

Your definition is missing the fact that branches have distinct names. Forks share the same name and are thus intrinsically ambiguous.

I've use a lot of different SCM systems and the experience of having a mysterious "fork" show up in your branch is unique to fossil as best I can tell.

anytime there is a git vs. fossil discussion inevitably forks are brought up as a serious weakness in fossil and git "wins".

I'm sorry, but that sounds like motivated reasoning to me. These people you're arguing with are Git fans and have found something that bothers you about Fossil and are using it to win the argument, even though it has little practical import. Working around the equivalent in Git is more trouble than in Fossil, so how can Git "win" on this point?

just run the below script which emulates two users making simultaneous commits.

That appears to be fixed with fossil rebuild --wal.

Richard, is there a good reason why WAL mode isn't the default for new Fossil repos?

The workaround aside, this looks like a bug. Shouldn't these forks be prevented by the checkin transaction, regardless of the journal mode?

This is why I thought mattwell must have either been running without autosync enabled or had a 3+ level deep cloning hierarchy; both insert communication barriers between the repos, allowing forks. Forking shouldn't be possible with a master-and-clones arrangement, if autosync is enabled.

I disagree that it is purely motivated reasoning. Forks are a genuinely annoying disruption when working with fossil. Try to put yourself in the shoes of the user. The user has to stop and correct a problem that he or she did not create. And yes, the fork is a problem and it must be resolved before work can continue. Even with the auto merge fork feature the process of resolving a fork is not really trivial. Stash any local edits, fossil up trunk, fossil merge, potentially have to resolve merge conflicts, fossil commit, stash pop any edits.

I think this is only a significant challenge on busy repositories. It might be that fossil is simply not appropriate for repositories where there are twenty or more commits in an 8 hour work day.

Note that with Git the user who is about to create the fork is blocked from pushing until they resolve the fork in their own private copy of the repo. I find git very frustrating to use but in this detail it does have a better solution. The burden should be on the user trying to get their changes into the central repo to get things clean.

WAL mode won't work in a shared environment. I'm guessing that all you are seeing is that on your single user, single host system WAL mode is fast enough that you can't do commits quickly enough to cause forks. I think is may be a mistake to make WAL mode the default.

To properly mitigate forks fossil could put all new artifacts into a quarantine pool. Then it could process the artifacts and update the db and if it sees a fork the offending artifact is left in quarantine and a message issued. This would be data-safe and non-disruptive. It would be important that the person syncing artifacts that cause the fork gets a clear message that their sync is quarantined.

We do have autosync turned on and it is a star topology. Everyone connects to a single server. Try running the script I provided and you will get forks.

I did run the script, and I fixed the problem by adding the command above just after the fossil init step. I've tried it now several times with that line present and commented out, and the fix seems entirely reliable. Try it there.

with Git the user who is about to create the fork is blocked from pushing until they resolve the fork in their own private copy of the repo.

That's what Fossil is supposed to do as well in a star topology like yours, and it's why I characterize your finding as a bug, rather than a usability weakness. There's no good reason for Fossil to create forks with your script. The only way to cause them should be to force Fossil somehow: turn off autosync, use 3+ level cloning, or pass --allow-fork to fossil checkin.

The very fact that that flag exists tells you this is a bug. Fossil wouldn't need that flag if its normal mode of operation wasn't supposed to prevent forks.

WAL mode won't work in a shared environment.

WAL was created for shared environments. Specifically, multiple-reader single-writer concurrency, which is Fossil's primary mode of operation.

WAL is faster than regular SQLite journalling, so it's better as concurrency goes up. When it causes speed hits, it should amount to dozens of milliseconds, not something that matters when you only have dozens of write transactions per day.

To properly mitigate forks fossil could put all new artifacts into a quarantine pool

That's not far off of what WAL mode does already.

That replicates here. I expect there's just a missing transaction barrier somewhere in Fossil, or one that doesn't stretch far enough.

The user has to stop and correct a problem that he or she did not create. And yes, the fork is a problem and it must be resolved before work can continue.

I don't understand this part. Why does anybody have to stop and fix a fork? Why does a fork prevent forward progress? If you don't understand why the fork occurred, or feel like fixing it would be a disruption, then simply ignore the fork and wait for somebody else to fix it. Or leave it unfixed.

In Git, the fork must be resolved before a user can push to the central repository. There is no choice. That policy prevents forks (and disconnected heads) in the central repository, but forces users to resolve forks before committing their work. I don't understand how this is less disruptive than giving the user the option of resolving forks now or in the future, or not at all if that is what they want. From where I stand, Fossil gives you choices, whereas Git forces one particular policy.

Your request for a setting is essentially a request to turn off these options that Fossil provides and force a single policy on everyone.

Whenever you have multiple people working on the same project at the same time, forks are inevitable. The only way to prevent them is to disallow concurrent development. Fossil makes the inevitable forks utterly harmless, and should the developers wish to resolve a particular fork, they can do so easily and at the time of their choosing. Git forces a particular resolution strategy on forks, because it is unable to delay resolution without creating disconnected heads. Hence Fossil provides much more flexibility. Do I understand correctly that you consider this added flexibility to be a bug?

In Git, the fork must be resolved before a user can push to the central repository. There is no choice.

Another way to view this is that Git imposes a UNIQUE constraint on the heads of branches within a single repository. Fossil does not.

With Git, in order to sync two repositories, they must be aligned in such a way that branch heads are unique after the sync. Prior to the sync, it might be that the branch has multiple heads (a fork). Git requires that the fork be resolved prior to the sync.

Fossil places no restrictions on when two repositories can be synced. If there is a fork, it can be resolved before the sync, after the sync, or not at all, at the developers' discretion.

The constraint in Git arises from the fact that it is using a key/value storage system with unique keys. The name of a branch is a key, and the head of the branch is the value. Because keys must be unique, the heads of branches much be unique in Git. If the underlying DAG does not actually fit that pattern, then you have a disconnected head.

I've stored the flat xml libre office files in fossil for years. Though I haven't often contended with multiple editors of those files. I haven't looked closely, but when I have images in the docs (usually) the diff compression kinda goes out the window. It's like just storing multiple whole copies of the file. But, I continue, because I like the ability to add long commit messages and/or tech notes to commemorate the wherefores and the whys of what I was thinking when updating the document. -- the reason for the changes is usually different than the content of the doc, and often more important.

too bad more "file" formats aren't SQLite-based! (right, drh?)

I've been toying with document storage in sqlite where each element is a record, and storing the edit diffs via triggers... one day.

Though, I pretty frequently dump a whole SQLite db to text, and capture changes to that in Fossil... some kinda satisfying reflection/recursion going on there.

if there is no conflict: fossil commit -m "merged fork of tip1 and tip2" else

Automatic commits of merges are always, without exception, a Bad Idea. (Yes, i know that certain git workflows use these, and i consider such workflows are braindead by design.) Just because a generic algorithm can merge two arbitrary text files without an explicit conflict does not in any way imply that the merge makes any semantic sense whatsoever. That applies to source code as well as non-sources like XML documents, both of which may, after a merge, still be syntactically legal but semantically nonsense.

You and i are both working on file foo.c. You change the signature of the function bar() while i, in another location entirely, add a call to bar() using the current (from my perspective) signature. A marge might even compile, depending on the nature of the change, or maybe the change is now semantically incompatible: you swapped the order of two args which have the same type, so that my usage of (x,y) is now supposed to be (y,x).

Cats and dogs living together, and all that.

Edit: my apologies if "braindead by design" comes across as a personal attack. It's not intended that way. Rather, it's an attack on the seemingly widespread practice of auto-committing of merges.

when I have images in the docs (usually) the diff compression kinda goes out the window

That should depend solely on the image file format, which then brings you to my image file format vs repo size experiment.

I realize that LibreOffice must be doing Base64 or similar encoding in their flat XML files for binary data, but that shouldn't change the point of the experiment. Small diffs on uncompressed data in Base64 should delta-compress just as well as with the original binary data.

too bad more "file" formats aren't SQLite-based! (right, drh?)

While I'd be happier to see SQLite as the basis of aggregating file formats than Zip, it wouldn't change the conclusions from my experiment. A PNG in a SQLite file is just as bad as a PNG in a Zip file, to Fossil.

Automatic commits of merges are always, without exception, a Bad Idea.

I think that is wrong. I fully agree that automatic commits of merges are almost always a bad idea, but there are often exceptions to a rule.

Here we are discussing unintended divergence. The users are expecting commits to be sequentially on the same branch on the timeline.

Note also that I suggested this be an optional feature. If you don't have a problem with forks or if you don't mind having angry users poking their head in your cubicle because they just wasted a hour trying to figure out why central automated testsuite runs failed to pick up their commit then by all means don't turn on the feature.

Note that in any modern team doing continuous integration and testing the bug you described should be caught by functional and regression tests before it hits a release. Fossil bisect would be used to find the offending commit and the fix developed.

In short, if you want to scale in terms of team size and commit rate you are almost certainly going to have some auto-commit in your integration and testing automation. The fork behavior of fossil interferes with this kind of automation and having an automatic solution would be very helpful.

[paraphrasing] Forks due to overlapping commits in a well connected network environment cannot be prevented.

Why not? In a simple star cloning topology, all three repos needed to cause the current problem are in communication simultaneously, via the central repo. That central repo is in a position to reject the fork-creating checkin, as long as --allow-fork is not given.

#2 is a training or message issue

I agree. The only case I can imagine where passing --allow-fork is a good idea is when Fossil is being run as part of some automated process, and its creator prefer that the script creates a fork than to take the time to write code to handle the fallback case. (e.g. fossil up then retry until you succeed.)

The "or use --allow-fork" bit should be removed from src/checkin.c. Leave the option in the program, and show it on --help, but don't tell interactive users to use it. It's bad stock advice; it guarantees forks. With that gone, Fossil will only tell them to update, which is the right stock advice for interactive users.

#3 also cannot be easily prevented.

Certainly, and that is the only sense in which I agree with drh's statement that forks are inevitable in concurrent development. If you have repos that aren't always in communication, forks are, indeed inevitable. That's another way of stating my prior observation that Fossil is an AP system. To prevent that under every possible condition, we'd have to make Fossil a CA system.

CA is out of the question. That amounts to a requirement that all remote parties be online before any commit can be allowed, so that everyone can agree on every commit. No way. That's the worst of all worlds for distributed and centralized version control.

That leaves CP, and I've been thinking about this since suggesting that Fossil might be able to switch into a CP mode for both commits and locks, given enough servers in communication with each other. For locks, CP mode is a hard requirement, but for checkins, I remain ambivalent about the idea. My current opinion is that it would be fine for it to be an option for those who prefer voting-based consistency over speed of checkin. I think I'd turn it off, and let checkins remain AP. I'll take the tiny risk of forks, if the alternative is slow commits every time.

I have received tickets that say "I committed my changes but user X can't see them!".

Up-thread, you claimed this happens "0.1%" of the time, which I found hard to believe, so I created a simulation to test it. With the default values — 20 checkins per day, ~3-second commit collision window, 8 hour working day, normally-distributed random checkin times spread over most of the working day — the chances of it happening are actually closer to 4%.

In perspective, that means it will tend to happen about once a month, under those conditions.

This is just a simulation, so everyone's actual working environment will differ from it. But, play with the values to make it match your own typical case and see what it tells you.

This is a JupyterLab notebook requiring the R kernel, using only core R packages. It's checked into Fossil as www/collisions.ipynb.

Add a feature "autoforkmerge"

I'm with Stephan: you're trading one problem for another. Granted that the new problem has a lower chance of occurrence, but now the times it occurs are hidden until they pop up from concealment and bite you, possibly far in the future from the actual merge point.

It might be nice to have such a feature as an option for the rare users who want it, but since forks are, in principle at least, entirely avoidable in a case like yours, I don't see the need to add it anytime soon. Let's fix the collision problem instead, okay?

(aside: how the heck do you do verbatim in markdown?

For short inline snippets, with backticks.

For multi-line segments, you can either use GFM-style triple backticks or the HTML <pre> tag:

verbatim text, GFM-style

verbatim text, HTML-style

See the Markdown formatting rules document, part of every Fossil installation, present in the hamburger menu in the default skin.

EDITS: Merged a double quotation of the parent post, fixed a misattribution, and did some clarity tweaks.

In perspective, that means it will tend to happen about once a month, under those conditions.

That is actually pretty dang close to what we are seeing for total forks. I was just assuming that most were due to #1 or #3. Nice work on the modeling.

That leaves CP, and I've been thinking about this since suggesting that Fossil might be able to switch into a CP mode for both commits and locks, given enough servers in communication with each other. For locks, CP mode is a hard requirement, but for checkins, I remain ambivalent about the idea. My current opinion is that it would be fine for it to be an option for those who prefer voting-based consistency over speed of checkin. I think I'd turn it off, and let checkins remain AP. I'll take the tiny risk of forks, if the alternative is slow commits every time.

IMHO if you all were able to pull this off it would be a huge win, making Fossil ideal for everything from the lone programmer up to good sized multi-faceted teams dealing with a mix of code and binary data.

See the Markdown formatting rules document, part of every Fossil installation, present in the hamburger menu in the default skin.

Yup, I did read the formatting rules - I just didn't see this "begin each line with at least four spaces or one tab to produce a verbatim code block."

I don't see <pre> or triple tick in the 
rules page
either. But the four spaces works - even if it
is tedious to use.

Can't use tab as the browser takes that and moves focus to the Preview button.

I just didn't see this "begin each line with at least four spaces or one tab to produce a verbatim code block."

I've expanded the document to discuss all of this.

Can't use tab as the browser takes that

It does work for embedded documentation, which can be in Markdown format.

The "or use --allow-fork" bit should be removed from src/checkin.c.

I've done this. I hesitated to do it on trunk without getting permission first, but I believe strongly in this change. If another core committer (especially drh) disagrees, back it out.

Nice work on the modeling.

Thanks, but I find it all rather dodgy:

1. The 3-second commit collision window is based on my end user experience, from Carriage Return on a fossil ci command to getting the next command prompt. Even if that number's accurate as a benchmark of the ci command, it probably isn't the actual size of the collision window.

2. It doesn't model your remote users. That would be two Gaussian distributions with different means, overlaid, with little to no overlap. Assuming your 20 commits per day are an aggregate from both locations, you'd have to split the cps value over the two (or more?) distributions.

3. It assumes that two back-to-back commits are always from different users. In my experience, that's not a great assumption. I make multiple back-to-back commits myself. I am unlikely to interfere with myself.

Bottom line, beware that this simulation might just be feeding your confirmation bias. It might feel right, but is it true?

Hi,

Just to put my two cents. I have been using Fossil with people without any previous direct experience with any DVCS whatsoever (journalist, librarians, researchers, students, etc.) and none of them have found any issue with fork as an unintended branch terminology. So I think that the bias of judging "proper" behavior to Fossil from Git exists and when people have not any prejudices the Fossil terminology is easy to understand and apply on a common workflow with the projects we work with (with a small community, sharing most of their time zone, but the last part is pretty irrelevant, terminology wise).

Cheers,

Offray

I disagree.

It makes things harder for (inexperienced) users who "just want to get things checked-in", and learn about the nature of forks and how to resolve them later.

Not sure, but I might have given up using Fossil, thinking it's too complicated, without this helpful hint.

anytime there is a git vs. fossil discussion inevitably forks are brought up as a serious weakness in fossil and git "wins". that sounds like motivated reasoning to me

I only had basic experience with git before I "discovered" Fossil. Fossil's documentation was the first I heard of "forks", at least in the meaning of "unintended branches". (Previously, I knew "fork" as in "InkScape is a fork of Sodapodi" - that is, someone taking a project in a new direction for whatever reasons. So, even github's use of "fork" was confusing at first glance.)

While I understand how forks happen, I have always been of the opinion that as soon as a (potential) fork is detected, the receiving server should reject the incoming commit.

This has nothing to do with git. My use of git has only been very basic - just enough to get the latest code, commit my changes and (for github projects) make a pull request. This thread has been the most education discussion on git I've had.

Recently, a fork even happened while working on Fossil itself, because I was not able to keep sync'ed "in real time" -- a scenario, for which Fossil has been explicitly designed, in the end.

I didn't want to "switch my focus", and resolve the fork, but concentrate on my own work, get things checked-in, and turn to the merge conflicts later.

The use --allow-fork hint was welcome, even to me, I don't know all these switches by heart.

Note also that I suggested this be an optional feature. If you don't have a problem with forks or if you don't mind having angry users poking their head in your cubicle because they just wasted a hour trying to figure out why central automated testsuite runs failed to pick up their commit then by all means don't turn on the feature.

That seems to me a loaded statement - your users should either know the value of understanding their tools, or they should be working to documented processes, which should include the possibility of forks and what to do. If they can talk to you they should be talking to their colleague who checked in the other side of the possible or actual fork. They should know what the automated test tool does if there is more than one open leaf on the branch it is testing. Why aren't they keeping open browser tabs of the timeline of their local repo and of the one it syncs with, and refreshing them before and after checkins. I do that, and it's just me by myself.

OK, that was probably a bit loaded too. But I don't see that there is anything wrong with forks per se, and nothing anyone has said here has convinced me otherwise.

Complaining about forks seems to me like trying to solve the wrong problem.

"unintentional branch" would be no clearer to me than "fork", and both cases would require me to go look up/ask about the meaning

But, looking up "fork" will then require looking up "branch". So, "unintended branch" saves one look up.

Also, phasing out Fossil's current meaning of "fork" would benefit Fossil in the medium to long term by removing confusion with more common uses of "fork".

(I do NOT want Fossil to be git. For one, I dislike git. Unfortunately, git (and github) dominates the DVCS (and VCS) arena, so its terminology is more understood. Even if new comers to DVCS have no trouble understanding Fossil's meaning of "fork", it opens them to later confusion. Also, new comers are more likely to be be exposed to git before Fossil.)

That seems to me a loaded statement - your users should either know the value of understanding their tools, or they should be working to documented processes, which should include the possibility of forks and what to do. If they can talk to you they should be talking to their colleague who checked in the other side of the possible or actual fork. They should know what the automated test tool does if there is more than one open leaf on the branch it is testing. Why aren't they keeping open browser tabs of the timeline of their local repo and of the one it syncs with, and refreshing them before and after checkins. I do that, and it's just me by myself.

Yes, in a small team a fork is merely mildly annoying. With a large, fast moving team the disruption from a fork can be costly. Part of the problem is that forks are not frequent. With about a fork a month it is probably a different person every time that must "fix" the fork. In this same repo I do fossil up several times a day. I have to make a point of checking the output because from my perspective "fossil up" is not a reliable starting point to getting to work. I have to verify that there was no fork.

I really don't know how to convey the impact of this. No, it is not a deal breaker but after having used fossil for quite a few years I'm at a point where the next time someone proposes going to git I'm going to be open to the idea. I'm stating for the record, in a high speed, intense development team with dozens of users in three timezones the irritation of forks is a serious factor.

Yes, you can mitigate with training.

Yes, in a perfect world everyone would have full situational awareness.

Yes, fixing a fork is easy.

In my opinion based on many years of supporting it, this problem is an important point of consideration for anyone looking at using fossil, especially in a corporate environment.

the irritation of forks is a serious factor. ... Yes, fixing a fork is easy.

There's a certain degree of cognitive dissonance there.

Forks in git are apparently, based solely on what i've read about them, highly dreaded, and nobody likes dealing with them. Fossil, you admit, makes handling of forks easy.

You're willing to accept git's "dreaded" handling for forks, despite the pain everyone seems to agree they cause, but not willing to accept fossil's "easy" handling for forks because of "the irritation".

Obviously, you've already made up your mind that fossil forking is a Really Bad Thing, and certainly you're entitled to that opinion. i suspect, however, that most of us opine that forks in git are Even Worse Things.

FWIW, i've been using fossil regularly since Christmas break of 2007 and have yet to have a problem with forks. Yes, i've had unintentional forks, but the resolving them is absolutely trivial.

Sidebar: unless i'm sorely mistaken, it was two commits from Richard and myself within one second of each other on Feb. 8, 2008 which caused the first-ever unintentional fork. My commit was 1 second after his and i didn't notice that it happened (fossil didn't warn me, due to a race condition), but Richard quickly noticed, after which he wrote a lengthy mailing list post about the experience. i'm going to quote the bottom part of that mail because it's currently somewhat relevant:

--------------

    From drh Feb 8, 2008, 6:40 PM

    ...<big snip>...

    What happened was that Stephan's fossil instance check to see
    if his version was a leaf and found that it was.  It then
    proceeded with the commit.  However, in between the time that
    Stephan's fossil instance checked to see if it was on a leaf
    and the time when the commit actually occurred, my fossil
    instance had done another commit.

    I do not see how this problem can be easily fixed.  Perhaps it
    is not important to fix it - how often do two developers commit
    within a second or two of each other?  And when an unintential
    fork does occur, it is easy enough to merge the branches back
    together to fix the problem.

--------------

That last sentence has been proven time and again to be the case.

(i unfortunately don't have a mailing list archive link to that post - it was dug out of my gmail box.)

I should refine what I said.

as soon as a (potential) fork is detected, the receiving server should reject the incoming commit

That should be "upstream repository" instead of "receiving server".

Better yet, the downstream repository, before pushing, should pull, then check for forks. If no forks are detected, then the manifest push to the upstream repo, which should then check for forks. If a fork is detected, reject the rest of the push and discard the received manifest. Otherwise, accept the rest of the push.

Since the commit(s) about to be pushed already refers to its parent, finding siblings is a simple SQL query. Then examine each sibling to find potential forks.

Note that this will require that a repo containing un-pushed commits push them in timestamp order to be sure that no descendants of the fork commit are pushed.

There's a certain degree of cognitive dissonance there.

Forks in git are apparently, based solely on what i've read about them, highly dreaded, and nobody likes dealing with them. Fossil, you admit, makes handling of forks easy.

No cognitive dissonance. The difference is crisp:

For fossil: A fork potentially affects everyone actively working and someone, usually not the creator of the fork, must stop what they are doing and fix it.

For git: The user who would be creating the fork must address it before their changes get into the repo.

There is something supremely irritating about being stopped from getting your work done by an unrelated action by someone else.

Pragmatically a fork is no big deal.

Emotionally a fork is a big deal. You are in the flow and getting things done then out of nowhere you are disrupted completely out of the blue. This is the factor that leads to pissed off users.

The problem arises because the pull, commit, push cycles can overlap such that interleaving of commits can happen. One fix that would work would be to have an option "commit-lock" that does the following:

The commit starts by setting a flag at the server side "commit-in-progress". The pull, commit, push is completed and then the commit-in-progress flag is cleared.

Obviously you'd have to take care of abandoned commits and other real life stuff but the basic idea would bring the ACID back into the picture.

Or maybe make it:

" or use --branch" to start a new branch.

After all, a fork is a form of branch and a named branch is easier to work with than an effectively nameless branch.

If I had seen the "--allow-fork" hint before I read about the possibility of forks, I would have stopped to find out what it meant, so focus already switched.

As for how I would have reacted if Fossil was my first VCS and saw that, I don't know. By the time I stumbled onto Fossil, I was already well versed with SVN and a few other VCSs and was experienced with using branches.

To my thinking, would be better to steer the user toward using proper branches if she/he chooses to defer updating.

Having developers work on different branches then resolving overlapping changes during an integration phase is a good way to avoid forks.

Still, having the upstream repo reject attempts to push commits that would introduce forks is something that Fossil should support. Much as I prefer Fossil to git, this is one thing I think git has done right (even if not for the right reasons).

Stephan's fossil instance checked to see if it was on a leaf and the time when the commit actually occurred, my fossil instance had done another commit.

That explains what happens with mattwell's symptom repro shell script. Where I got mislead is that I thought there was a transaction lock on the central repo while all of this was going on. That is:

1. Remote clone A starts a checkin, doing all of the steps in drh's 11-year-old email with the cooperation of central repo C.

2. By the act of A checking whether a checkin is allowed on C, C starts a SQL transaction that is only committed or rolled back when the Fossil-level commit succeeds or is rejected. It's a 1:1 arrangement. (I suspect it isn't 1:1 today, which is the problem.)

3. Remote clone B comes along while that transaction is open and tries its own commit, but is blocked because of the open transaction.

4. A's SQL transaction/Fossil commit (overlapping things, in this scheme) is accepted.

5. B gets told that it can't commit because it isn't at the tip of the branch, so it needs to update first.

This puts the burden of resolving the problem on one of the two committers: the one who came along second.

If it happens that A wasn't at the tip of the branch when it opened the transaction, it gets rejected, its SQL transaction rolled back, and B's commit is accepted, assuming it was at the tip of its branch at the time it started the commit.

Bottom line, I don't see why a race condition is necessary in a star topology. C is in a position to arbitrate the problem.

I think I see why Fossil does still have this problem: CGI. These conversations between A, B, and C are taking place across multiple instances of the Fossil binary in the case of pure CGI, or across multiple children of the fossil server in that case. Because of that, a SQL transaction can't remain open across conversations, because when the Fossil child/CGI instance dies, its SQLite conn closes, rolling back any un-committed SQL transactions.

Recently much praise was heaped on Fossil's ability to do CGI, so this may indeed be an intractable problem, where we get to choose one of two praiseworthy features to have, because we can't have both.

EDIT: That suggests a path out of the tarpit: use 1:1 SQL/Fossil transaction commit mapping when Fossil is running in server mode, so that the SQLite conn can remain open through the whole process.

use 1:1 SQL/Fossil transaction commit mapping when Fossil is running in server mode

In other words, instead of the parent process just forking off a child and have that child behave the same as in CGI mode, on receiving an HTTP connection, the parent handles the initial HTTP negotiation through the point where the remote process is identified by cookie or auth params.

If the fossil server parent believes this is a new conn, it proceeds much as currently, except that the parent creates the SQLite conn and starts a transaction on the repo DB. The parent then stores that SQLite conn in the parent's session info list.

If the parent believes a new conn is from a previously-connected client, it looks up its session info and passes the prior SQLite conn down to the child. If that conn was used by a previously-forked child to start a SQL transaction, that transaction will still be open for this new HTTP client instance.

It is up to the parent process to close the SQLite conn, and it only does that on getting SIGCHLD when it sees that there are no open transactions on the DB conn. I couldn't quickly find a way in SQLite's C API to detect this, but it looks like a check on db->nStatement == 0 would do what I want here.

There's a possibility of SQL conn abandonment in this scheme, but that's easily dealt with using session timeouts. If a conn is left open for, say, 10 minutes with no child re-grabbing it, drop it and let its transactions roll back.

All of this aside, why does this problem occur when the Fossil binaries are running standalone, as with mattwell's shell script symptom repro script? There's no CGI involved, and I don't see why there would be any forking involved.

And now that the option is removed...

The --allow-fork option isn't removed. All my change does is cause fossil checkin to not recommend passing it as stock advice for solving the problem. You can still find the option documented in fossil help ci and you can still pass it.

Good plan. I've just checked in such a change.

On moosefs - WAL mode crashes.

We left this detail hanging. I suspect MooseFS is not implementing file locking or syncing correctly across machines, which prevents SQLite from providing its ACID guarantees. This is a common enough problem with SQLite on shared network filesystems that the standard advice is, "Don't do that."

I'm curious why you'd want to do that anyway. Why put a distributed version control system on a distributed filesystem? Surely that's redundant. Fossil already gives you a way to distribute checkins across machines. Put your served Fossil repos on a local FS and let Fossil sync changes across machines in its standard way. Then you can use WAL, which should reduce the chance of a fork, per my earlier tests.

If you need your Fossil repos to be on your MooseFS SAN for some reason separate from Fossil's DVCS nature — e.g. backups — those repos can be pull-only clones, so they cannot create forks. Those clones can have WAL mode turned off.

why does this problem occur when ....

Because it has nothing to do with CGI. It is due to the non atomic sequence pull, commit, push. Add a table lock with fields branch, lock_state, repo_id. Add calls get_lock, release_lock and figure out how to call them before the pull and after the push and the forks will be gone. Pardon the terse description. Out walking the dog :)

Yes, I saw that the CGI issue and the two-step "at branch tip" and "try checkin" sequence are unrelated after posting.

Your scheme would indeed solve the problem for all three cases: standalone, CGI, and fossil server.

File locking works fine on MooseFS, journal_mode delete and persist work great on MooseFS. WAL mode seems to trigger some other bug or limitation. My intent is to point out that WAL mode has cases where it is not safe. Please don't make it the default.

Sharing fossils via file:// across MooseFS, NFS etc. works great. Consider it a really nice feature of fossil. In cases where IT prevents users from starting servers, file based sharing is wonderful. BTW, on NFS file:// is much, much faster than ssh:// or http:// and it has been more reliable when committing really large commits.

Odd, I thought I'd logged in when I posted previous post.

On additional reflection I think per-branch locks are not necessary. Even on our busy repo the forks are only one or two per month.

flow would be something like:

success=fossil pull -get-commit-lock
if success
  do the commit
  fossil push -release-lock
else
  sleep a little
  try again or if waited over give_up seconds
     go ahead and do the commit
     fossil push

I'm still unhappy with that.

• Fossil is a very flexible tool, allowing flexible work-flows.

• Fossil has no problem with forks.

• The fact that --allow-fork must be specified explicitly already makes it clear that this is not the common case.

• Hiding/obfuscating features from users in an attempt to influence their decisions seems utterly wrong. A much better approach may be to provide enough documentation to help users make their own decisions.

In the end, I'm happy you haven't completely patched out the --allow-fork option ;-) But please bring back the useful hint about it!

Also, phasing out Fossil's current meaning of "fork" would benefit Fossil in the medium to long term by removing confusion with more common uses of "fork".

You mean the thing you eat with? :-)

By analogy with that, but more generally, a fork is where something that looks like a line divides and continues as two lines. The term was used in a Software Configuration Management context well before there were any distributed "version control systems" - in reference to branches.

It is ridiculous to talk about phasing out a perfectly sensible and appropriate use of a common word just because the word has also been used for something else (for which it is also appropriate).

The more I think about it, the more questionable I find your approach.

There's a feature (forks) you don't like, or that doesn't play well with another feature you're reasoning about (locks).

Instead of educating users about the problems with and alternatives to forks, you're trying to make it harder to discover (and remember) a feature that may be very handy in certain situations, and you're hoping users won't discover it "too early", if they need to look it up in --help, explicitly. It's like trying to force all users to work the way you prefer.

A transparent and enlightened approach would be mentioning both --allow-fork and --branch in the warning message. There may even be a hint that --branch is preferred, but simply dropping --allow-fork seems not correct.

Or, are you looking for a "global solution" (don't use forks, anybody, please) to a "local problem" (locks don't play well with forks)? ;-)

Fossil has no problem with forks.

But people do, giving what is probably the single biggest thread on this forum so far. :)

Hiding/obfuscating features from users in an attempt to influence their decisions seems utterly wrong.

This is not "hiding" anything. This output also doesn't include any of the other dozen or two options that also apply to fossil ci. Are those functions "hidden," too?

The point of this message is to give advice on moving forward. I, and others, believe --allow-fork is a bad idea under every possible case, for interactive users. Therefore, Fossil should not use any part of its one-line message area to recommend its use.

No one is stopping you from giving this option or recommending its use. By my checkin, I am saying that I think it's a bad idea and that Fossil should not recommend this as stock advice to interactive users, because it's almost always the wrong advice.

But please bring back the useful hint about it!

Nope. Someone else can back it out if they think my choice here is wrong, but I will not.

Instead of educating users about the problems with and alternatives to forks

That's funny...you're arguing against my change which recommends that people use --branch to get out of this trouble, rather than --allow-fork, yet somehow we're against educating people about forks and their alternatives?

mattwell's right: when you create an unnamed fork, every Fossil instance that syncs with that repo immediately starts carping about it, placing the load on fixing it on everyone. Whereas if you use a branch instead, no one is inconvenienced or bothered. What's not to like?

Personally, I think unnamed forks should only ever occur when you've got someone working with autosync disabled or behind a 3+ level cloning scheme so that all repos aren't always sync'd. It's a method for Fossil to keep moving itself forward without blocking progress, not something that end users should be making use of directly.

A transparent and enlightened approach would be mentioning both --allow-fork and --branch in the warning message.

I think it's important to keep this message short, but even that aside, I refuse to recommend an option that I think should never be used interactively. That option is, IMHO, useful only by software tools that must bull forward blindly when this happens rather than get stopped.

Nope. Someone else can back it out if they think my choice here is wrong, but I will not.

For what it's worth, my vote goes for Florian, but i'm not emotionally attached enough to the --allow-forks hint to fuss about it either way. i completely fail to see how forking in fossil is "a bad idea under every possible case" (or even a majority of them), given how easy forks are to resolve, but, based on the feedback in this thread, i may well be in the minority.

This output also doesn't include any of the other dozen or two options that also apply to fossil ci. Are those functions "hidden," too?

Just the options to deal with the forking situation are interesting for the warning message, i.e. "fossil update", "--allow-fork", and "--branch".

... yet somehow we're against educating people about forks and their alternatives?

In my opinion, good education is never about hiding facts (i.e. options). Never. ;-)

I refuse to recommend an option that I think should never be used interactively.

"Refusing to recommend" is not the same as "actively removing a long-standing helpful hint". ;-)

Branches also have their problems, with their double-nature as propagating tags on one hand, and "special tags" on the other. If everything is moved to branches, and branches are freely renamed, this may cause trouble, as well.

A fork may simply be the preferred approach, for some people, in some situations, and I see no reason why to withhold useful information from these people.

good education is never about hiding facts

In principle, I agree, but since I am not going to download a copy of the Schimpf book and open it in Acrobat on every error in the name of complete education, we have to cut it off somewhere. I think one-line errors are a good choice. Longer errors tend to be ignored on tl;dr grounds.

"Refusing to recommend" is not the same as "actively removing a long-standing helpful hint". ;-)

Since the error message is recommending a course of action, removing text absolutely is the same thing as refusing to recommend!

Look: you've got checkin rights, so if you want it changed, you revert it, and you can support the thing. I'll drop out of the discussion.

By the way, please click the Reply button on the message you want to reply to. This is at least the sixth time in the past week or so that you've replied to a message by clicking on a nearby Reply button, which screws up the threading.

i completely fail to see how forking in fossil is "a bad idea under every possible case"

I'm not saying that forking is bad under every possible condition, I'm saying that manually creating forks by using --allow-fork is a bad thing, always.

Automatically created forks are fine. These include several cases:

1. autosync is disabled

2. you're using Fossil in a truly distributed fashion, so the sync state of the repos drifts over time, as syncs occur, allowing multiple commits to a branch tip without warning

3. you're writing a bot that does things to Fossil with little error-handling ability and so would prefer to just blindly add --allow-fork to the ci commands to ensure that checkins are always accepted rather than being lost.

These cases are all inadvertent forks created by dumb software, where the solution must be left to the end user to work out.

But when the option is given to an interactive intelligent user, I can't see any good reason for them to purposely create a fork, which is what happens when you give this option. They're giving a big F-U to the rest of the users of that repo. "I can't be bothered to work out the merge, someone else deal with it." And that's what Fossil now does: it starts griping at every other user about the unresolved fork. Why is that now everyone else's problem?

This is repo pollution, and the dumper needs to be fined and re-educated.

I think one-line errors are a good choice.

Definitely:

Error: would fork. "update" first, or use --allow-fork or --branch.

There's even room for the word "recommended", somewhere ;-)

Look: you've got checkin rights, so if you want it changed, you revert it, and you can support the thing. I'll drop out of the discussion.

I'm not going to do that, for plenty of reasons. I'm just not happy with your check-in, which does nothing to resolve the greater problem in any way, but just makes it harder to find ONE of the possible solutions.

By the way, please click the Reply button on the message you want to reply to. This is at least the sixth time in the past week or so that you've replied to a message by clicking on a nearby Reply button, which screws up the threading.

As soon as I need horizontal scrolling in my web browser to be able to read the posts, or when a new "subtopic" is started, I prefer replying to the parent message. But I will stop that, sorry.

This is repo pollution, and the dumper needs to be fined and re-educated.

Well, this is one way to see it (I see it differently, given how easy it's to deal with forks in Fossil). But this needs communication and common sense, instead of stripping options from warning messages.

Error: would fork. "update" first, or use --allow-fork or --branch.

Instead of continuing to bang the drum trying to get me to rewrite this message for you, convince me that creating unnamed forks interactively is a good thing. To me, forks are accidental damage that the one with the brain has to fix because the brainless software failed and chose to create a fork rather than lose information. An interactive user is presumed to have a brain, and so has no excuse to be perpetrating the same kind of problem.

Until I am so persuaded, I will remain adamant that Fossil should not recommend this as a solution to the "would fork" problem.

your check-in...does nothing to resolve the greater problem in any way

Sure it does: it removes mattwell's issue #2 from the table, leaving only issues #1 and #3 to be dealt with:

#1 is the race condition issue under discussion already, up-thread.

#3 is autosync off, which is outside the discussion as far as I'm concerned, since this whole thread is about interactive usability, and forks created under this condition are created automatically by Fossil as a coping strategy during sync.

horizontal scrolling in my web browser

I also find the depth of this thread annoying, but it's even more annoying to have messages in the thread disordered.

And that's what Fossil now does: it starts griping at every other user about the unresolved fork. Why is that now everyone else's problem?

"Griping"? Maybe that's another message that needs streamlining. And it's not everyone else's problem, you see one of those messages, you see how close to your own work it is (quick look at the graph), and if it's close enough to worry you, talk to the person who created it. If that happens often, then you need to look at the project branching policy and areas of responsibility within the project. If a fork is too old (FSVO), tell its creator to close it, merge it, or turn it into a branch, you shouldn't have to do it for them.

This is repo pollution, and the dumper needs to be fined and re-educated.

Pollution? It's definitely a Git thing to want a neat repository. Fossil shows the truth, warts and all.

Well, this is one way to see it

It's exactly analogous to the negative externality of pollution: the perpetrator is creating a cost on every other user of the system in order to avoid doing a bit of extra work themselves.

stripping options from warning messages

You say that as if --allow-fork vs --branch were a neutral value judgement, that they should be considered equally. If the two were close in terms of costs and side effects, I wouldn't be arguing this with you, but they're a long way apart. Until you convince me that advising use of --allow-fork isn't likely to create more problems than using --branch does, I'm not going to budge on this stance.

... convince me that creating unnamed forks interactively is a good thing.

I can't, obviously -- but you can't convince me that forks (in Fossil) are evil, either. It's a matter of personal taste, and preference.

But this is not the question, somehow. My request to bring back the --allow-fork hint is independent of any preference. I'm okay with an additional --branch hint added, but intentionally making --allow-fork harder to discover (and remember), instead of giving better hints to ALL options, looks like... like you don't have any convincing arguments, either? If --branch vs. --allow-fork were so clear, you wouldn't need to remove it?

"Griping"? Maybe that's another message that needs streamlining.

The message Fossil gives when you've got an unnamed fork is telling you something important: you might not be checking your changes in on the branch you think you are, because the current branch's name means more than one thing at the moment. Thus the negative externality: someone else couldn't be bothered to either merge their change into the branch properly or push their change off onto another named branch, so now everyone else working on that branch now gets forked.

you see how close to your own work it is

That's not how Fossil forks work: they set up a situation where it isn't immediately clear to the end user which of the two forks their change will land on. You end up with two independent lines of development, with some number of the repo's user's stuck on one side and some number on the other. Until the fork is resolved, the two lines of development will diverge indefinitely.

On that note, I don't see the problem with the current terminology: this condition in Fossil is exactly analogous to what happens when a FOSS project is forked.

you shouldn't have to do it for them.

You either fix it for them or keep ignoring Fossil's complaints about the fork on every checkin while working on that dual-tipped branch.

Fossil shows the truth, warts and all

Sure, but that's no cause to be recommending methods of creating repo ugliness.

Again, forks are fine for Fossil itself to create in order to avoid losing work, but for interactive use? Never, short of sync problems due to autosync off or 3+level deep distributed cloning.

you don't have any convincing arguments, either

I thought all of this was clearly understood already:

• It's bad for fossil ci to have an ambiguous meaning.

• It's bad for the development team to have their work split among the forks based solely on the time they happened to have last sync'd, which changes which branch tip is considered newest at the time they go off on one of the two forks. Worst case, it splits your team 50/50.

• It's annoying to see the complaint from Fossil that "a fork has occurred" on every checkin. (It's good that this diagnostic exists, but bad when it's triggered.)

If --branch vs. --allow-fork were so clear, you wouldn't need to remove it?

No interface is completely intuitive, not even the nipple. Part of UI design is training users in correct use, and that means guiding them away from uses that create problems.

but you can't convince me that forks (in Fossil) are evil

If by evil you mean harmful then it is clear that forks are harmful. A fork is intrinsically ambiguous and ambiguity is harmful in our context of source code management.

For the scenario where you get the would fork message is there any advantage to creating a fork over creating a branch? The branch is unambiguous and just as easy to merge as the fork. I could see an argument where a lone programmer might find the fork slightly less effort due to the existence of fossils ability to automatically merge forks but that is so slight and so specific a use case that it just isn't compelling.

It would be fair for the fossil developers to approach this with a policy statement such as "fossil strives at all times to eliminate or minimize ambiguity". Applying this policy to the fork situation yields see a fair argument for eliminating the --allow-fork switch entirely. As a compromise the --allow-fork feature is kept but it is not advertised in the help.

This single change of replacing --allow-fork with --branch in the message will save me personally some one or two trouble tickets a month and have the excellent benefit of exposing users to branching. Believe it or not there are still a few holdouts who resist branching and want everything done on the trunk.

Thank you for your time to reply to me. I get all your arguments, but still prefer forks over branches, in certain situations.

But what bothers me is different:

Fossil encounters a potential fork, and gives a list of options how to proceed: "update", "--branch", etc.

Users may have the impression that the list of options given here is COMPLETE, and may not even think of looking at fossil help ci for more information.

I'd be fine with:

Error: would fork. "update" first, use --branch,
or see "fossil help ci" for more options.

The help text could show two screens with arguments why forking is discouraged, any maybe a warning about an angry phone call by Warren whenever a fork is created ;-)

But giving the impression of a complete list, and actively and voluntarily suppressing information, hoping that users stay in the dark and won't find them, looks like "manipulation", or "censorship". This can't be the right approach.

... convince me that creating unnamed forks interactively is a good thing.

Here's why I prefer forks over branches for temporary splitting due to merge conflicts:

• All check-ins are on the target branch for which they are intended, and the /timeline?r= view gives a complete overview.
• No need to think of another branch name, no need to clutter the global branch (and tag) name space.
• As it's not so easy to rename nested branches in Fossil (see below), I try to avoid too many nested branches (i.e. avoid a new nested branch for each merge conflict).
• The steps to resolve a fork are identical to the steps to integrate a branch.
• One of the fork arms could still be moved to a separate branch, if required later.

If the goal of your efforts to "avoid" forks (well, hide the feature from users) is to guarantee the "uniqueness" of branch names (i.e. make sure each branch name resolves to exactly one single open leaf check-in), then consider the following:

• Branches may be renamed, causing name collisions with other branches.
• Branch names may collide with tag names.
• Users may forget to close (intentionally reused) old branches.
• Users may (unintentionally) reuse existing branch names.

These cases look more common than creating a fork, and Fossil would have to apply sharper restrictions to enforce the "one name"↔"one open leaf" policy, and would lose a lot of its flexibility.

Side note: To illustrate the problem with the renaming of nested branches, consider the following branch hierarchy and the corresponding tags (*=propagating, -=cancelled):

    ┌─● *branch=LEVEL2, *sym-LEVEL2, -sym-LEVEL1
    │
  ┌─●   *branch=LEVEL1, *sym-LEVEL1, -sym-trunk
  │
  ●     *branch=trunk, *sym-trunk
  │

Now rename the branch "LEVEL1" to "LEVELX":

    ┌─● *branch=LEVEL2, *sym-LEVEL2, *sym-LEVELX, -sym-LEVEL1
    │
  ┌─●   *branch=LEVELX, *sym-LEVELX, -sym-LEVEL1, -sym-trunk
  │
  ●     *branch=trunk, *sym-trunk
  │

It's not longer possible to refer to the (former) branch "LEVEL1" by name, as the new name "LEVELX" is now resolved to the (former) branch "LEVEL2".

I prefer forks over branches for temporary splitting due to merge conflicts:

Ah, an unstated major premise! I wish we'd had this out in the discussion before.

Neither I, nor those in any project I've ever been on, have used nested branches/forks for dealing with merge conflicts. We always either:

1. Resolve the conflict in the integration checkin. This is a useful strategy only up to a point: if the diffs get too large, then...

2. Merge the diffs since the branch creation point up into the branch — e.g. fossil merge trunk — then sort out the problems on the branch, so that there is no merge conflict when merging the branch down into its parent. (Example here.) When that step happens to be the penultimate step on the branch, the integration checkin can end up with zero diffs relative to the last checkin on the branch.

Whichever option you choose, all of the benefits in your bullet list are either included or irrelevant.

This also deals with your problems with nested branch renaming, later in your post.

If the goal of your efforts to "avoid" forks

My goal is to avoid bothering others on the project with my work, because they've got their own work to be getting on with, and vice versa. The most important forks to avoid are thus those involving either trunk or other long-lived working branches.

Forks on short-lived personal working branches don't bother me, but since I've disposed of the need for those above, that seems like a low-impact observation to me.

I've only ever used nested branching a few times in my career. Usually when I'm tempted by the idea, I find that it's better to make a separate branch from the parent of the current branch — usually trunk — for that feature, finish it quickly, merge it back down into trunk, and then merge that finished feature up from trunk into the branch I was on when I had the idea.

Branches may be renamed, causing name collisions with other branches; Branch names may collide with tag names.

While I agree that that these are problems, they seem like non-sequiturs to our discussion here. They neither argue for or against forks. Forks have the same problems, if you rename them, which you can, as you point out elsewhere.

Users may forget to close old branches.

Easily dealt with by occasionally sorting /brlist by Status.

That is also how I deal with a cluttered global branch namespace: only those that sort to the top are of ongoing interest. I try to keep that sub-list quite short in the projects I manage.

enforce the "one name"↔"one open leaf" policy

That's exactly the policy that forks violate. It is the very definition of a fork!

EDITS: Typos...

Forks on short-lived personal working branches don't bother me

If you're thinking that that's a good reason to bring --allow-fork back in the error output, realize that it's a fourth-level-deep allowance:

1. I run my own projects more or less like that of Fossil itself: almost all checkins are self-contained and so go on trunk or one of the long-term stable branches. Forks on these branches are bad for the reasons stated early in this sub-thread.

2. When I do need a temporary working branch, it's almost always short-lived enough not to require any of the gymnastics we've been discussing recently in this sub-thread.

3. When a branch gets complicated enough to require such gymnastics, I never reach for forks as a solution, per the prior post.

If I were to rewrite that error message in terms of forks, it'd be something really long, like

    fossil_fatal("would create an anonymous fork.  \"update\" first or use --branch to\n"
                 "create a named fork.  You can also use --allow-fork to bypass this\n"
                 "error, but you should not do that on branches that others are using\n"
                 "for long-term work, such as trunk.");

I've tried to keep that message brief, but that's already tl;dr for a Fossil error message, as far as I'm concerned.

It's bad for the development team to have their work split among the forks based solely on the time they happened to have last sync'd, which changes which branch tip is considered newest at the time they go off on one of the two forks. Worst case, it splits your team 50/50.

Where they land depends on which checkin their working directory is at when the move. If they have to do fossil open branchname they always end up on the most recent checkin of the branch, regardless of forks. If the working directory is already open and they do fossil update branchname, then

• if they are below a fork, the update fails

• if they are above a fork, they end up at the tip of the side of the fork that they are on

• if they are on a different branch, they end up on the most recent checkin of the branch

If they do fossil checkout mybranch, they end up on the most recent checkin of the branch.

fossil update always (I think) says if there is more than one leaf. So, by the way, does fossil status. But fossil checkout and fossil open (which just calls checkout anyway) do not. Probably they should.

Thank you for your time to reply to me.

I've spent some more updating the relevant documentation page, with the relevant portions starting here. (Also, check out the pretty new SVG branching diagrams! GIF p'tui! :) )

Users may have the impression that the list of options given here is COMPLETE,

I already answered that above: this error message also doesn't include the contents of fossil commit --help or the Fossil docs, or the Schimpf book, or the contents of a Google result for "fossil checkin". Every error message from Fossil is incomplete in some sense. We get to decide only where to draw the line.

Since the only justification I can come up with for interactive use of --allow-fork is 4 levels deep from normal operation, I don't think it makes the cut.

or see "fossil help ci" for more options.

Isn't that implicit?

What I could get behind is a URL to the above document. I don't think Fossil has done this in the past, but having a URL as the second line of every message doesn't sound like a bad thing to me.

Then we have all the space we want to present arguments and training.

Where they land depends on which checkin their working directory is at when the move.

You've forgotten the case where autosync is off or defeated by lack of a network connection. My documentation improvements (linked above) explain how the development of a shared branch can be forked.

What I could get behind is a URL to the above document.

I've done this. Can we end this debate now?

Also, if you've looked at the new documentation since the last post I made, it's been updated several times since.

Indeed, just because a file is in XML (or other textual representation), it doesn't mean that it can be merged like a source code file.

For instance, in a video game, levels could be represented as xml files, with one xml element for each scenery object in the level.

Level artist A fills up an empty space with a building. At the same time, level artist B decides to put a rock in that same empty spot.

The two modified files, having each added a different xml element to the file, would merge without any conflict. But now, your game level contains a rock and a building overlapping at the same location, which doesn't make sense.

The solution for this is lies in process and workflow, not on trying to let the computer automatically resolve the problem (merging), which it can't.

So you add a process rule: "only one person is allowed to work on a game level at a time", and indeed you need a locking system to support that process.

As it happens, I've just added a relevant test case to the Fossil self-hosting repo: the new www/branch*.graphml input files and their associated *.svg output files. If I change one of the labels on one of the nodes in the diagram — nominally a one-byte change — I get this result:

  $ fossil test-delta-analyze branch04.graphml x
  original size:     16516
  bytes copied:      16512 (99.98% of target)
  bytes inserted:        4 (0.02% of target)
  final size:        16516
  delta size:           59

That's not as small as it ought to be, but it's a pretty tiny delta artifact.

Now let's look at what happens on the output (SVG) side:

  $ fossil test-delta-analyze branch04.svg x
  original size:     18186
  bytes copied:      18180 (99.97% of target)
  bytes inserted:        6 (0.03% of target)
  final size:        18186
  delta size:           88

Still not awesome — it should be a 1-byte change — but it's much better than making the equivalent change in a tool that outputs GIF instead.

That's how XML should work in Fossil. Any tool that does significantly worse with equivalent-size changes needs to have a bug filed against it.

Where they land depends on which checkin their working directory is = at when the move.

You've forgotten the case where autosync is off or defeated by lack of= a network connection. My documentation improvements (linked above) ex= plain how the development of a shared branch can be forked.

I have forgotten nothing, that was a technical description related to the state of the local repository at the time the various actions occur.

If you have autosync working at the time, all that means is that the local fossil will know everything it needs to know to prevent or warn about forks and to allow the user to ensure they are going to work based on the right checkin.

If there is in fact a chain of syncs, then autosync merely gives the local fossil a better chance of knowing everything that is relevant.

I've spent some more updating the relevant documentation page, with the relevant portions starting here. (Also, check out the pretty new SVG branching diagrams! GIF p'tui! :) )

I have just been looking at that. I worry that some of it sounds a bit too much like opinion - though I can see that you've tried to avoid that.

For me, for a lot of the things that have led up to this, "the solution for this lies in process and workflow" (to quote a post by Zlodo in this thread).

If only there was an easy way to contribute changes to the Fossil documentation...

I really am done with this debate now.

All I'll say at this point is that it's far more worthwhile improving the documentation than fighting over what goes into an 80-column wide error message from Fossil.

I've done this. Can we end this debate now?

Sorry to extend this... I have limited time so when I get time I have to get my thoughts out when I can.

Personally, I would have preferred no URL in an error message. It probably belongs in the extended help, but what exactly is wrong with this:

would fork. "update" first, or use --branch, or --allow-fork

I understand that it might make the user think about what they are doing, but what's wrong with that? After all, they are using a command-line tool and not a UI. Once they have thought through it for the first time, they will simply ignore this warning and know what to do. Most likely they will either turn on autosync (why was it disabled in the first place because it's enabled by defualt), or they will get in the habit of pulling first, or they will realize that they can use --branch.

In any event, this is such a trivial matter that once they've done it they aren't likely to be concerned about it the next time they encounter it. Once they have a fork, then Fossil instructs them to use "fossil merge" so again, I'm not sure I see much value in suppressing the --allow-fork option.

Whether they use --allow-fork, or use --branch, they are still going to have to figure out how to get their code back into the branch from whence they came.

Thanks,

Andy

One other statement that I don't think is as accurate as it could be:

https://www.fossil-scm.org/home/artifact?udc=1&ln=185-187&name=d11887717c7a4273&txt=1

Sure, the forks that happened due to autosync being disabled entirely, or due to network hiccups can be considered accidental, but those that were intentional are not. So how can forking be defined to be accidental?

Thanks,

Andy

Sorry for my delayed answer, I needed a wider monitor, first ;-)

My thoughts about the "URL solution":

• The hard-coded URL in multiple locations has its maintenance cost (URLs may change).
• Following URLs from an error message seems less likely than reading fossil help ci.
• The information linked by the URL gives the impression that you are still trying to hide the --allow-fork option at any price. Fossil users are mature and responsible individuals, and I really miss the point, here!

/!\ Warning /!\ If you ever visit me, we'll have Spaghetti Bolognese. I'll hand you a branch, and hide all my forks, and I won't give you any hint where you can find them. More so, I'll REFUSE to give you a hint that I have forks at all. Even though you may be the best branch-eater in the world, I promise you'll desperately need a fork. Oh, and come in a white shirt ;-)

I worry that some of it sounds a bit too much like opinion

This is tutorial documentation, not reference material. It should have a voice and be opinionated. It's why every bit of software popular enough gets more than one book about it: because there's more than one way to teach the software's "proper" use. This document's starting premise can be stated, "We have branching for a reason, and you should probably be doing that rather than forking."

It said that before I got here. My late additions to the document amount to only about a third of the current material, and a lot of it is pure addition rather than replacement. This means a lot of what you're seeing was already there, and was written by a variety of users, mainly Jan Nijtmans and D. Richard Hipp, in that order. My check-in count on that file is currently fourth, behind Joe Mistachkin, though I'll probably pass him before the sun comes up today.

Roll back to the version from last week and re-read it, if you think this is all just my opinion. I'm mainly just expanding on what was already there.

Much of that expansion amounts to a distillation of this thread, which means "the other side" now gets more of a say than it previously did. The "Justification For Forking" section is entirely new. You think you're underrepresented today...

I think it better to accurately describe the Fossil fork terminology and how to deal with what it means than it is to include a value judgement here, or even use scary strong statements.

The bit you're quoting could be rephrased "Don't intentionally fork trunk or other long-lived shared working branches." What's wrong with that stance?

I've tightened the prose up to be clearer about the qualifiers, but it still says essentially the same thing.

Unintentional forks are dealt with elsewhere in the document.

automatically merge in a fork

Wow, I didn't know that was even a thing. I've added it to the document.

I consider this short command further proof that Fossil does have an opinion on forks: that they're bad, to be avoided as a general rule, and fixed as soon as possible. (The prior version of branching.wiki said much the same thing.)

A branch is a fork that is named differently than its parent

Isn't that just a more pedantic way of saying "named"? Why name anything if it is not different from its parent? Even a son named after his father at least gets a "Jr." or a Roman numeral. We consider it humor when a dog called Dog.

The document already deals with propagating symbolic names, down in the section on "Tags and Properties."

are not all branches intentional?

Yes, and most forks are unintentional, by this document's philosophy, and I believe in practice most of the time. I suspect even the most ardent forking fans in this thread have more accidental forks than purposefully-created ones.

Still, I've added a paragraph after that Key Distinction bit to clarify that a fork may be intentional. I think that's pedantry, given the rest of the document, but whatever.

I've just had the time to review your recent series of check-ins in detail. And I have to say I completely agree that "the solution ... lies in process and workflow", as mentioned elsewhere on this topic.

It seems we all have different habits and preferences. For example, I (personally, on my repositories) wouldn't commit a series with multiple small corrections and eliminations of minor spelling mistakes directly to trunk, but merge it from a separate branch when it's ready, to make it easier to review the trunk in "logical units". And you wouldn't use forks.

But I wouldn't add an --allow-trunk option, and hide it from the warning message, to make it harder for people to commit directly to trunk. So I'd also expect you not to remove the useful hint about the --allow-fork option.

This is in no way a personal attack on you or your way to work, nor a cynical remark, but an attempt to show that controlling user behavior by holding back information causes frustration.

This is tutorial documentation, not reference material.

Okay, but warning messages should not have a "tutorial character"?

the onus of dealing with committing content into a DVCS is on the users committing it

In another post, you throw shade on Git users who use it without as much knowledge of the tool as they ideally should have, but the only way I see Fossil as being immune from that condition is that Fossil is generally easier to understand and work with, so overall, a Fossil user probably knows their tool better than a Git user.

However, the underlying laziness still exists in the species. I don't think recommending a potentially harm-causing option as a quick fix is a good thing. That caters to the "Just make it stop hurting" desire, but if it's more likely to cause different pain down the road, I don't think it's a good thing.

If you understand what --allow-fork does and its consequences, then by all means, use it as freely as your good judgement demands. (And if you don't understand it, that's now covered better than ever in branching.wiki!)

My change to cause fossil ci to recommend --branch instead isn't susceptible to the same problem. You can't just pass that option blindly to make the pain stop, because unlike --allow-fork, it requires an argument. That in turn leads those ignorant of branching to the docs, which raises that user's knowledge level. Blind use of --allow-fork will not immediately teach the user about forks.

what exactly is wrong with this: would fork. "update" first, or use --branch, or --allow-fork

Because that feeds the "Just make the pain stop" impulse in the ignorant, while most likely causing more pain later.

Once and hopefully for the very last time: the option isn't gone, it's still there in fossil help ci.

I suspect if we instrumented Fossil with telemetry to count the number of uses of --allow-fork vs --branch, we'd find a clear preference for the latter, so why should --allow-fork get equal billing? Even putting it on the same line in the error, after --branch, is overselling its importance. I believe it's a rarely-used option, and that's as it should be.

You've got checkin rights, too: you want it reverted, you go right on ahead, and I'll give up.

What I'm not going to do is be bullied into trying to please everyone.

I understand that it might make the user think about what they are doing

That's the problem: it doesn't make the user do any thinking at all. "Oh, I can pass this option and Fossil will stop griping at me? Great! Shut up, Fossil!" Now they've got two problems.

That sentence was checked in by drh, not by me, so you can argue with him, if you like.

But also, that sentence refers to Figure 2, which is preceded immediately by, "Suppose two programmers make independent modifications to check-in 2. After both changes are committed, the check-in graph looks like Figure 2:"

That's what defines Figure 2 as an accidental fork.

URLs may change

That's something you worry about on the open web, where all web sites are not under your control. That's not the case here. If something under www/ in the Fossil source tree moves, the source code referring to it from src/* can be updated and checked in at the same time to fix it.

Fossil has a longstanding policy of renaming files only at great need, in part because there may be external URLs referring to the current name. Example: src/wiki.c isn't all about the wiki feature any more, but it's still called that, which misleads people spelunking for all of the code related to the forum feature.

Another reason for not renaming files is that it makes tools like fossil finfo fail, because they're not smart enough to trace back through a rename. File artifacts in Fossil are keyed by the full path name in the repository, so when a file changes name, most file-name based queries in the Fossil code will no longer find all of the artifacts. This policy in the Fossil self-hosting repo avoids confronting that weakness of Fossil.

I have a different policy on renaming in my own Fossil repos, and it causes a lot of trouble because of all of this. I choose to accept this pain because I'm anal about my files being labeled according to their content. But, I can get away with that, because even my most popular public Fossil repo has only a tiny sliver of the amount of Google juice as fossil-scm.org, much less that of sqlite.org. When I break an external URL, almost no one cares.

Following URLs from an error message seems less likely than reading fossil help ci.

Most terminal emulators have a right-click "View URL" feature these days. The primary exception being Windows, but that's just as it ever was.

And again, we don't need to explicitly say "Type fossil help $CMDNAME" in every error message. That's implicit. This URL points you to something you might not find on your own. I suspect there are a bunch of people reading this thread who found branching.wiki for the first time just this week.

Overall, I think the Fossil docs are underutilized, which is why we should be linking it into Fossil help and error output more often.

you are still trying to hide the --allow-fork option

...while at the same time documenting it better than ever before. Weird, that.

Will you please either revert my change or give up on this battle?

Because that feeds the "Just make the pain stop" impulse in the ignorant, while most likely causing more pain later.

We can do a lot of other way more harmful things with computers ... so no worries, here ;-)

I suspect if we instrumented Fossil with telemetry to count the number of uses of --allow-fork vs --branch, we'd find a clear preference for the latter, so why should --allow-fork get equal billing?

Rarely used options also need reminders, maybe even more so. I'd even drop the hint to --branch, and assume users would have added it already, if their explicit intent was to create a new branch, or would already have used fossil branch new.

The primary exception being Windows, but that's just as it ever was.

I have it! Windows! And the terminal detects URLs! But I prefer more focused work-flows without too many additional browser windows.

...while at the same time documenting it better than ever before. Weird, that.

With the hint about --allow-fork from the warning message, you know at least what to look for. From the linked anchor of your URL, the (apparently directly from hell black magic) expression --allow-fork does not even appear on the first page on the screen! This is obfuscation, and not a neutral short reference.

Will you please either revert my change or give up on this battle?

Yes, give up.

I admire your vehemence to define opinions and work-flows for other people ;-) You have "outforked" me ;-)

It said that before I got here.

I don't think it did.

Roll back to the version from last week and re-read it

I have been familiar with the contents of that document for quite a while (until a few days ago :-) ), and I don't think it has the same feel to it anymore - that's subjective of course. We all read in the light of our opinions as well as, more obviously, writing in that light.

I am leaving myself a note to read it again in a few weeks, to see what I think after the dust has settled (including the dust in my brain).

Eric

I might fork less if there were (is?) a way to see the a timeline of multiple specific branches at the same time.

+1

This would indeed be handy and I've had users request it before.

or see "fossil help ci" for more options

Isn't that implicit?

Surprisingly, no.

As you have pointed out in another post, people often are lazy, so if you don't remind them that help is available, they "forget".

Anyway, I think 'or see "fossil help ci"' is better than 'or use --allow-fork'

I see many reasons that Fossil is technically superior to alternatives. So, I care about Fossil's credibility in the collective eyes of the wider DVCS community.

Encouraging users away from an option that introduces forks is part of that credibility.

While I see good reasons for Fossil to tolerate forks in a repository, I also see good reasons for Fossil to (by default) inhibit pushing forks to other repositories.

Currently, we simulate this by wrapping Fossil in scripts and by having autosync set to pullonly. For "fossil ci", our script does:

system 'fossil ci $*';   # will auto-pull, but not auto-push
sleep 1;
system 'fossil pull';    # attempt to get any simultaneous commits
my $forks = `fossil leaves -m`;
$forks =~ s/^\s+//; $forks =~ s/\s+$//; # strip leading and trailing space
if length $forks { warn "There are forks: $forks\n"; }
else { system fossil push; }

(From memory. Probably forgetting some details.)

Fossil should have this capability built in.

I can see your points - in your local repo.

But then, how to prevent forks being pushed from your local repo.

One way is to always commit your work to a personal, preferably private, branch. Making the branch private will stop Fossil from pushing it's contents until you publish the branch.

Alternately, it looks like Fossil has a command hook (see https://fossil-scm.org/index.html/doc/trunk/www/th1-hooks.md) that could check for possible forks before allowing the push.

But, if auto-sync is on, this would only work if if gets triggered for the auto-push that is part of the commit command.

It seems reasonable to me that the command hook should be run for each of the automatic commands as well as the overall commit command.

not some random fork that might exist in the same branch upon which you are working.

It's not random, it's time-based, due to Fossil's AP nature: not everyone sees the same view of the repository at all times. Fossil sacrifices consistency in favor of availability and partition tolerance. Everyone can commit at any time, regardless of sync state.

Fossil can split a team's checkins across forks like so:

1. User A creates a fork. Doesn't matter how or why.

2. User B checks in sufficiently late that they see user A's checkin as the latest on that branch, so her checkin goes in as a child of A's latest checkin.

3. User C updated his clone at "step 0" above then went off-network or set autosync=0. There was no fork at that time. Now he's still off-network and needs to make a checkin, so it goes in against the later checkin on that branch than A forked from in step 1. If User A didn't do what he did, there would be no fork: C's checkin would be in a linear line along the development of that branch, even though he's offline at the time. User C later comes back online and syncs, giving that branch a later checkin than either A's or B's.

4. User D comes along, sees C's latest checkin on the other side of the fork, so now her commit goes in against that side, being the latest.

The end result is that A and B are on one side of the fork, C and D on the other, and everyone else who comes along after this will get put on the fork that just happens to have the latest checkin at the time they join in this circus.

That is if either user A or B happen to make a commit at step 5 above, User E coming along later will get put onto their side of the fork instead of user C and D's side.

I don't believe "fossil ci" has any ambiguous meaning in the presence of forks

I mean "ambiguous" in the way of my story about Users A thru E. You can reconstruct history from the Fossil timeline to explain why it occurred, but some of these users couldn't predict what would happen in advance without going out of their way to carefully work it out up front.

Consider the plight of a new User F, coming upon the repo in this state. He can predict what will happen only by:

   $ fossil clone https://example.com/forked-repo.fossil fr
   $ fossil timeline -R fr     # discover which branch tip is newer
   $ fossil open fr            # end up on that branch tip now

or equivalently:

   $ cd /path/to/checkout/of/fr
   $ fossil pull
   $ fossil timeline
   $ fossil update forked-branch

That is, it is only by inspecting the timeline between the clone/pull and the subsequent open/update that User F can predict which side of the fork he'll end up on.

If User F doesn't inspect the timeline upon the local clone he won't be able to predict, a priori, which branch tip he'll end up on. If he checks in /timeline on the remote clone via HTTP instead, there's a race condition involving sync time to his local clone.

rather than ping pong in the source code.

That's what I was saying with those repeated offers: I will not revert your change, I will take being overridden in the code as "I lost the argument."

Instead what happened is that several people argued me into ping-ponging myself. I don't see a meaningful distinction between the two cases.

If I'd known this was going to be so controversial, I'd have put my work on a branch, but I thought it was obvious and straightforward, so I didn't.

(In case anyone's wondering why I gave up eventually, it's because Stephan Beal holds a bigger veto than any of you, in my mind. When he weighed in, that was the end of the argument, as far as I was concerned.)

User B checks in sufficiently late that they see user A's checkin as the latest on that branch, so her checkin goes in as a child of A's latest checkin.

Which should say:

User B moves (update, checkout, or open) to the branch A's fork was on after the fork was created and then the move will see user A's checkin as the latest on that branch, so her checkin goes in as a child of A's latest checkin.

If "current" in B's working directory was already on that branch her checkout will be the child of that "current", and forks or the "latest" don't come into it. (She may still be potentially creating a fork of her own.)

I've created a swimlane diagram showing this problem and added it, along with a better version of the above explanation, to the branching doc in a new section, "How Can Forks Divide Development Effort?"

this is very helpful I'd say. thanks. this explains nicely the real problem: it is not the fork in itself but the possibility of different users unwittingly ending up and checking in on different "branches".

question 1: regarding a "minimal example", checkin 2 could go away from the diagram and explanation, no?

question 2: structurally, forks are just unnamed branches, right? is there anything in the internal logic of fossil to prevent treating them exactly the same way as named branches? because obviously the only real problem with fork vs. branch is that if the fork where a "real" (named) branch in your diagram, all users would continue to check in on trunk and checkin 3 would just hang around there waiting to be dealt with by somebody.

I believe all the reasoning in favour of branching vs. forking is only concerned with this difference: for a fork the divergence you describe in the diagram can occur (different users unwittingly on different lines of development) while it cannot (unwittingly) with branches.

so again: is there any principle hurdle in just treating forks as "true" unnamed branches so that a user never moved away "secretly" from the branch he is supposed to be on in the first place? could not fossil be modified accordingly? or what would be the problem with this?

checkin 2 could go away from the diagram and explanation, no?

Sure, but that would make Charlie (previously called User C) responsible for everything, because he created the fork by being offline. I want to make it clear that this can occur without any one user being wholly responsible.

In the current text, it looks like Alan is the primary responsible party, but it could be that he was also offline, in his case between check-ins 1 and 2, so that when he came back online and pushed check-in 3, Betty and Darlene see that check-in before doing their work, but Charlie doesn't because he went offline after check-in 2 came in from some other user. Now the consequences are no one's sole fault.

forks are just unnamed branches, right

Yes, as is made clear in that same document.

is there anything in the internal logic of fossil to prevent treating them exactly the same way as named branches?

It is the fact that branches are named that is the problem, not the fact that forks are unnamed. If humans were willing to type hex strings at their computers instead of tag/branch names, none of this would be a problem. Humans, as ever, are the problem. :)

When I say fossil up trunk, Fossil has to go and figure out what "trunk" means at that exact instant, and it might run into a race condition that makes it ambiguous besides. It does its best, but what it's giving me isn't strictly speaking "trunk," but instead [abcd1234...]. I think it gave me trunk, but that's because I'm unwilling to think deeply about hex strings and DAGs on each interaction with Fossil.

It is the fact that branches are named that is the problem, not the fact that forks are unnamed. If humans were willing to type hex strings at their computers instead of tag/branch names, none of this would be a problem. Humans, as ever, are the problem. :)

When I say fossil up trunk, Fossil has to go and figure out what "trunk" means at that exact instant, and it might run into a race condition that makes it ambiguous besides. It does its best, but what it's giving me isn't strictly speaking "trunk," but instead [abcd1234...]. I think it gave me trunk, but that's because I'm unwilling to think deeply about hex strings and DAGs on each interaction with Fossil.

which the user ideally should not need to too often :).

still: topologically (regarding the DAG at a certain point in time) the fork is not different from a proper (named) branch point. so forking per se seems unavoidable in certain situations but how further checkins are handled principally is under the control of fossil. I understand that currently both routes (to use a different term than 'branch') after the forking are considered to be part of the same branch and location of further checkins in the DAG is determined by chronology of the checkins alone. but that seems to be rather a matter of convention/mental representation than being a true distinction from the 'two distinct branches' picture': it seems to be equally valid to consider the forking point as an ordinary branching point (and possible assign a reserved branch name to the forked route to discriminate it from the mother branch) and to to consider the forked route as being just another branch (no longer belonging to, say, trunk).

so my question remains: what deep reason is preventing fossil from treating the forked route mostly like any other (true, named) branch regarding what is happening to further checkins from different users to the repo? meaning: all your checkins go the the current branch/route if you do not explicitly request otherwise. thus preventing the divergence (users unwittingly end up on two different "routes"). is this technically unfeasible or difficult or prinicpally problematic to do?

Thank you -- that is outstanding! I had a feeling that there was a way to do this and that I had simply not figured out the syntax yet.

I could imagine a version of the brlist page with checkboxes that would allow a user to select some number of branches and then click a button to get the filtered timeline of just those branches. On the other hand, I think I will be quite happy to type the URLs out by hand myself, as you say.

Are you suggesting that given the two options, "update" or "--allow-fork", your colleagues are choosing the latter? Why don't they choose "update"? Neither require more effort than the other and "update" has fewer characters to type and is generally easier to spell (at least on qwerty). Especially given that fossil really only requires you to type "fossil up" to get it accomplished.

I'm guessing it is mostly an education problem. The team is large enough that people come and go. Fossil is essentially unheard of and every new user must be trained and we are far from perfect in getting that training done for everyone new. This is made worse for us by the fact that some users come in from a git background and I suspect that fork feels less dangerous to them than update. Does update clobber my local changes? Without training a new user can't be sure. To make things even worse everyone is under intense pressure to get things done. There's a release today, and tomorrow and ... you get the picture, so users don't have time to stop and research this obscure SCM tool "fossil" that they are forced to use. All this combined leads to the occasional use of --allow-fork. I eventually get notified that there have been a few forks and I dig through the timeline looking to figure out who dun it. I then contact that person and do a little one-on-one training. Hardly the end of the world but a completely unnecessary disruption for all involved.

Mostly I prefer "update" but on occasion I am wont to use the evil "--allow-fork".

As has been exhaustively discussed I think if you are a lone programmer or you have autosync turned off and resolve the fork before syncing then sure, use of --allow-fork is slightly less work than simply using --branch current-branch-name-fork-reason (that is my fork-branch naming methodology when I want to avoid update).

However, in any other situation where there are other users who will be affected I believe it is unfair to create the burden of a fork. No matter if you are a team of experienced users and routing around the fork is not a big deal, it is still a disruption.

Branching is cheap, easy, and unambiguous and fossil has excellent resources for managing divergence via branches.

Forks are intrinsically ambiguous and can be confusing, disruptive and time consuming (dunno about you but for us that time == $$$$). Interesting thought, if a good engineer is costing the company $100-$200 an hour and each fork causes 2-10 people a half hour of confusion and lost work then do the math. All this for a "feature" that is of benefit to a few lone programmers and could easily be done away with.

I guess it falls to the fossil core team, optimize for the lone programmer and promote --allow-fork or optimize for larger chaotic teams and don't promote --allow-fork. Going further and adding a lock around the pull, commit, push cycle would make fossil resilient for use in mega-corp teams :)

My swim lane diagrams have no branch names. The problem can occur without them.

In my diagram, users Alan, Betty, and Charlie do not use branch names explicitly at all. They're all working from the legacy of the last "open" or "update" command they gave to Fossil, at some indefinite time in the past. From that point forward into my narrative, Fossil on these three user's machines is working with checkin IDs, not with branch names.

User Darlene does give a branch name on switching to this forked branch, either explicitly (fossil up forked-branch) or implicitly (fossil open ~/museum/fr.fossil giving trunk). But again, it doesn't matter what the branch is named. It'll happen with all labels stripped out, too.

You seem to want to flip the problem around, having Fossil implicitly apply labels to fork points. But what would they be, if not checkin IDs?

So cool. I now see under "advanced" the tag filter. I've been shirking my duties in keeping up with all the great work you all do. Thanks!

still: topologically (regarding the DAG at a certain point in time) the fork is not different from a proper (named) branch point. so forking per se seems unavoidable in certain situations but how further checkins are handled principally is under the control of fossil. I understand that currently both routes (to use a different term than 'branch') after the forking are considered to be part of the same branch and location of further checkins in the DAG is determined by chronology of the checkins alone. but that seems to be rather a matter of convention/mental representation than being a true distinction from the 'two distinct branches' picture': it seems to be equally valid to consider the forking point as an ordinary branching point (and possible assign a reserved branch name to the forked route to discriminate it from the mother branch) and to to consider the forked route as being just an other branch (no longer belonging to, say, trunk)

If Fossil knows it's going to be a fork at the time of checkin, it tells the user (with the message containing or not containing "--allow-fork" as per the discussion above). If Fossil does not know at that time, it will know after the next sync - which involves two repositories, which one of them should do something? And the users who created the two checkins which are the start of the fork both thought they were checking in to the branch. There is no way that Fossil could tell which checkin should get a new name and thus not be on the branch anymore. If it choses one somehow, that a whole different kind of confusion for the users. If it uses the same name every time, that's another source of confusion, and if it makes one up using it would be as bad for the user as using the hex IDs.

The answer would be worse than the problem (which some of us don't believe in anyway :-) ).

I think I understand all that... maybe I fail to make myself clear...

next try. the point I want to make is basically simply this: if, at the point where a fork is going to appear (at the "update or use --allow-fork" message in the past) your user A actually would do a "ci --branch FORKBRANCH" followed by "up trunk" all other checkins by A and B-D would end up on trunk while that checkin no. 3 would sit there on branch FORKBRANCH waiting for somebody to take care of merging it back into trunk. I believe that was what you where advocating earlier in this thread: "use a 'true' branch rather than allow the fork", right? so this would be a more or less sane/easy situation. which is good.

OTOH, if the same DAG topology for checkins 1-3 is produced by a fossil-type fork, further checkins by the different users end up on different "routes" as you explain in your diagram. which is not so good.

question is: why cannot fossil decide to handle the forked route internally as a true branch distinct from trunk (or whatever the mother branch's name) and thus put all further checkins on trunk just as it would do if checkin 3 really had been explicitly put on a different branch?

am I missing something obvious here?

see my answer to the previous post by wyoung: I am wondering why it cannot (or should not) be enforced that fossil treat the fork checkin just as if it would be a ci to a new (named) branch?

the DAG looks the same for both scenarios. the difference is in how fossil interprets/treats the two topologically identical DAGs afterwards.

if it is a "real branch" that had been created instead of the fork and all users remain on the "mother" branch (say trunk) all their further checkins naturally go to trunk.

if it is a fork, the checkins are ending up on the two different routes "offered" by the fork, depending on chronology considerations. which is equivalent to a situation where users would be silently moved to a different branch... which is not desirable.

so what is fossil principally preventing from doing away with this discrimination between fork point and branch point as far as the decision where to put the next checkin is concerned?

I am sure this would require work (not sure how much, though) to implement. but is it principally impossible or undesirable? am I being stupid? not seeing the obvious?

what is fossil principally preventing from doing away with this discrimination between fork point and branch point

The distinction is human, not technological. If the fork point is named, we call it a "branch." If not, then it's a "fork." Fossil itself only uses differing terminology to cater to us weak humans.

(Okay, okay, that's not strictly true. Branch renaming doesn't work that way, for one thing, but the details of that don't affect the answers to your questions, so we can ignore that here.)

not seeing the obvious?

Okay, let's try it this way. Let's say branch names don't exist at all. Somehow, through some magic you have yet to specify, Fossil knows which side of the fork to use every time, right?

So, let us say for the sake of argument that the tip of what we used to call "trunk" back when Fossil had branch names is currently forked on some repo we're trying to deal with. What then does this command do:

  $ fossil open my.fossil

? Which fork tip do we get in our checkout directory?

If you say "the newest one," that's great. But what if the other branch tip was newer when faux user Alan gave the same command 10 minutes ago?

Now both you an Alan have a checkout of the same repo in your local directory, but your checkout contains the state of one fork tip, and Alan's has the state of the other tip. When you make your modifications and check them in, Fossil knows what revision is in each local checkout directory, and that becomes the parent of each of your checkins. Your checkin is against one fork tip, and Alan's goes against the other tip. How does Fossil avoid this, magically figuring out what you meant and so avoid the fork?

You can't prevent forks from occurring in a proper DVCS. The only way to avoid them is to fully centralize it, or as good as, by requiring that every clone be online and so they can all coordinate each checkin. Technically speaking, that's CA mode operation: consistent and available to those within the quorum, but intolerant to partitions, which means the "autosync off" feature has to go away. (That's one form of network partition.) It also means checkins while the network is down are no longer possible for the same reason.

The names are the problem here. It is the fact that we want to call both tips of trunk "trunk" that creates the ambiguity. If we humans would just be specific and say "[abcd1234]" instead of "[ef89ab12]" on every checkout or branch switch, Fossil could get its work done without any confusion at all. We humans and our weaknesses are the problem here. Fix that, and Fossil's problems go away.

EDITS: Typos and clarity.

at the point where a fork is going to appear (at the "update or use --allow-fork" message in the past)

--allow-fork is not the only way forks can happen. See the first two items in the Justifications for Forking section.

And keep in mind, a huge chunk of the argument here is people saying they don't want --allow-fork to go away, so you can't just dismiss that anyway.

We can't get rid of forks. We have to accommodate forks when they happen, whether intentionally or accidentally.

why cannot fossil...put all further checkins on trunk

Because that's not where the user was when they made the checkin. Their checkin was made against a particular fork tip; that's the only place it's been tested, so it shouldn't just be moved automatically by Fossil to the other fork tip.

That would amount to automatic cherrypicking and checkin, and we've already rejected automatic checkins of automatic merges in this thread, for good reason. This is a particularly bad sort, because there is nothing in Fossil constraining the amount of development between two fork tips. Someone could show up in a sync 2 weeks after a fork point with two dozen checkins made offline. Why is their fork tip suddenly "right" because it's got a newer tip than the old tip?

Automatic checkin of automatic merges only sound good to people with a lot more faith in Fossil's ability to figure out whether it's a good merge than it deserves. There is no AGI in Fossil able to reliably say "This is good, check it in." All it can detect are outright conflicts; a lack of conflicts is not the same as "It's good!" There are other ways for a merge to fail, some of which we've already discussed in this thread.

See the example about game level files in the XML files sub-thread, for example.

Although I've written messages in favor of automatic XML merging working with Fossil, I never suggested automatic checkin. A human still has to check the result manually somehow before checking it in. All I'm saying there is that it's possible for Fossil to do automatic merges with XML, as long as the program producing and consuming that XML acts sensibly. The human still has to check Fossil's work, and ensure that it doesn't confuse the program consuming the XML file.

This is also why we've previously argued against mattwell's idea of automatic fork merges: again, no AGI in Fossil, hence no faith that the automatic checkins won't create new problems.

I challenge the notion that Fossil will ever make a commit [to] the child of any other parent than the that one that I currently have checked out.

I've understood that abstractly, but I've missed a consequence of it: the race condition mattwell is talking about isn't necessarily a bug in Fossil, it's the race condition between his Bash script's fossil update command and the immediately following fossil commit command. That window means the commit doesn't necessarily go to the tip of the repository, causing Fossil's "would fork" message.

I've been assuming for some reason that the autosync in fossil ci would automatically fast-forward your checkout to the tip of the branch if you weren't already, but of course it won't, nor should it. I wonder if mattwell or his users is also assuming Fossil does this, or should.

That race condition is why Fossil has the "would fork" warning in the first place. If they're passing --allow-fork to get past it, then they're not only shooting themselves in the foot, they're doing it by aiming first, so they lose my sympathy.

Which would bring us back to the wish to remove this advice from the error, if I could be bothered to resume the fight. Which I can't. Someone else can fight it now.

Incidentally, none of this has anything to do with my fable, which shows what can happen even without tricky race conditions. It shows everyday battles over "Is my check-out's view the tip?" I've updated it to be clearer about the ways each user gets to their view of the repository as laid out in the story, most clearly in the numbered list of Betty's paths to her view.

To clarify, I'm saying that I no longer see any reason to suspect a second race condition in Fossil itself: the one in the shell script is enough to explain everything.

it's the race condition between his Bash script's fossil update command and the immediately following fossil commit command.

The auto-pull and following commit in fossil commit has the same race condition. The window might be smaller, but there's still a race condition.

To clarify, I'm saying that I no longer see any reason to suspect a second race condition in Fossil itself: the one in the shell script is enough to explain everything.

Look at the last version of the test script I posted. The lock is directly wrapping the fossil commit command. Comment out the lock calls, run the script and you'll get lots of forks. Uncomment the lock calls, run again, no forks.

I had this same thought the other day... basically what you're asking for is that the Fossil client, when it encounters a "would fork" situation, instead of returning an error, simply generates a random GUID and automatically forces that commit onto a branch? The branch could be something like <branchname>-GUID.

something like that, yes (followed by `fossil up trunk' to put the user immediately back on the mother branch). but I was mostly concerned with the consequences w.r.t. further checkins by other users (so that the checkins remain all on the same "route"/branch rather than diverging into two directions).

regarding the concrete behaviour in the case when the eminent fork can be avoided (when one gets the 'update or --allow-fork' error), the error message should be maintained (with whatever phrasing) since avoiding the fork altogether (by first updating) is definitely preferable. but if the fork is confirmed by the user, yes, than the sort of branching you describe above should occur. I believe the problem with forks mostly has to do with what wyoung describes in his wiki text: different users end up unwittingly in "parallel universes" and that does not happen if the fork is treated just like any ordinary branch.

from the answers I received (but have not thought through so far ...) I've got the impression that the above somehow is logically unsound/impossible. but is it?

] My question is, would that be any better than the current forking model? I'm inclined to believe that this would lead to just as much frustration as anything.

just silently branching, yes, definitely. but making a fork, when it actually happens (automatically or after user confirmation) behave like any other (named) branch w.r.t. to how the location of further checkins in the DAG are determined I presume that would be preferable: further checkins should never become childs of the forked checkin if someone is not explicitly updating to that specific checkin first.

I prefer forks over branches for temporary splitting due to merge conflicts

I assume you have auto-sync set to pull-only?

And that you are resolving the fork before pushing?

still trying to hide the --allow-fork option

What other DVCS has some form of this option?

I assume not git. I have searched (Google and others) but my search-foo seems not up to the task.

Without setting up experiments, I don't even know how git or any other DVCS detect or complain about forks.

Even if the --allow-fork option is hidden, Fossil is still much easier to use than git.

Even if the creator of a fork is diligent enough to make sure it never gets pushed, there are still good reasons - far beyond git's allergy to forks - for strongly discouraging creating them.

And since a fork can occur accidentally, there are good reasons to contain it as close as possible to the user that unwittingly created it.

there are still good reasons - far beyond git's allergy to forks - for strongly discouraging creating them.

Can you please help me to better understand your concerns by providing details of those good reasons for discouraging forks?

And I have to say I completely agree that "the solution ... lies in process and workflow"

Ideally, yes.

In the real world, forks happen and someone is forced to deal with them. All too often by someone other than the creator.

But I wouldn't add an --allow-fork option, and hide it from the warning message

Many popular applications hide advanced options from warning messages. It is generally considered good practice to do this. While I admit that it is inconvenient for me (in the apps I am an advanced user of) to have to search for those options, I agree with this practice.

--allow-fork is an advanced option.

Advertising options that make it easy to create forks just adds more to the problem of dealing with them.

I see nothing wrong with intentional use of --allow-fork

As someone who has had to deal with forks created by other people, I do see the wrong in any use of --allow-fork

While I understand the thinking behind the option, any commit done by an automated tool (except maybe one done by the developer's build tool) should go on a new branch to allow for proper review and testing before merging it in to the originating branch.

It might be easy to deal with forks, but proper branches are easier to deal with and no one gets unknowingly side tracked. Also, they make it clear that the creator knew they were creating it.

Can ping-pong between forks be reduced by expanding on the blockchain idea? The fork with the most "votes" wins. Only use the "latest" to resolve between equal length checkin chains.

The fork with the most "votes" wins. Only use the "latest" to resolve between equal length checkin chains.

Still doesn't deal with the underlying problem. And the more commits to either side of the fork only makes merging it back in harder.

That implies that it's safe to rebase a checkin against another tip of the forked branch automatically. I never want Fossil to automatically commit something against a basis that it was never tested against, and after you reflect on it a bit, I hope you also don't want that, either.

No, the solutions are a) avoid creating forks in the first place through policy and technology; and b) heal unnamed forks quickly when they do occur, since if we wanted them hanging around, we'd have named them, which would make them branches, not forks.