Seems like a simple hack for sure. I wish I wouldn't have to do that mainly only so that I don't have to remember to do it every time fossil is updated.

have you had any other unintended consequences from doing that? Does that only effect the temporary file created expressly just for doing diff's?

If your local checkout has a change relative to the upstream repository, updating that checkout will leave that change undisturbed unless the upstream version has changed that same line or lines sufficiently near to it to cause a merge conflict. Short of such a merge conflict, it is possible to carry a local change forward for a long time without explicitly re-applying it to each new version you update to.

I wish I had some way to evaluate whether this change should just be accepted upstream, but I don't have strong enough insights into the various dialects of "diff format" to make a decision to apply the patch myself.

Until someone makes such a decision, I see no reason why you wouldn't just maintain this locally, since it's solving a real local problem for you. There's nearly zero effort involved.

So enlighten me a bit.. what is the best way to maintain a "fork" of fossil? I clone the repo, create a workspace and edit that file and commit it to my own repo.. probably on a branch.....but then what?

I would not want to be building all the latest changes that are going into the fossil mainline on a day to day basis...., I would want to only update my repo when they release a new version of fossil, then I'd want to make sure to merge the right thing into mine and keep using my modified version of diffcmd.c... So what would my procedure need to be on the day they release fossil 2.9 and how do I make sure i'm working on a branch for now that is based on the 2.8 release of fossil, ignoring changes after that?

This does seem like something that ought to work the way you have done it to begin with, but that's just me. However working through this will also help me learn fossil a little more...so I should try that for a while..

Suppose your new branch is named "dewdman".

To merge the latest trunk changes into your branch do this:

fossil up dewdman
fossil merge trunk
# Resolve any merge conflicts
# compile and test
fossil commit

Since you do not have "push" permissions on the main repository, you should probably do this:

fossil setting autosync pullonly

Do that just once and Fossil will only try to pull changes from the main repository, and never try to push back your new check-ins. This will spare you from getting lots of warning messages.

Thanks for the pullonly recommendation..

One question about the pullonly setting.....

I will clone the fossil-scm public repo to my server machine here....

Fine so far. Except I have a cluster of a few machines.. the server is linux NAS and the other machines are mac and windows. So for my own repos I have a "server" repo that is on the linux machine. I run the server deamon there to easily get access to ticket tracking at a moment's notice more than anything. Then on each mac/pc I usually have cloned repos from that server...and I open workspaces on them to work as needed. They are autosync'd with the repo on the linux NAS...which is kind of the master for my own purposes and gets backup to the cloud, etc.

Now if I clone the fossil-scm public repo... I would need to create another clone on each of the Mac/PC's, so that I can build them there for each platform. Ideally I'd make the code change I want to do, on my server repo..(which is actually a clone from the public repo). and then they would pick up any changes with auto sync from my fossil-scm repo clone... Is that even possible, first of all?

But if I change auto sync to pull only on my server repo...will that negatively impact the auto sync that I need to happen between that server repo here and my various client pc/mac's?

Am I doing this all the wrong way and there is a better way?

The autosync setting is normally changed from its default on a per-repository basis only. You'd have to go out of your way and add the --global option to make it apply to all repos. Therefore, your setting for this on your local clone of fossil-scm.org/fossil has no bearing on autosync behavior in your other repos.

This option isn't currently carried from one repo down into its clones. If you say fossil set autosync pullonly on the Linux NAS, the Mac and Windows boxes will retain their autosync setting defaults on a per-repository basis until you change each one.

If you create a local-only branch on the Linux NAS and clone from that repo to the Mac and Windows boxes, they should each see the local changes. They can remain autosync on because they're sync'd with the NAS, not with fossil-scm.org/fossil. Automatic pushes from them will therefore go to the NAS, not bypass it and try to modify the Fossil self-hosting repo. You could therefore create a local-only branch on the NAS, clone it to the Mac, check it out there, hack on it, check in a change which autosyncs to the NAS and stops right there because the NAS has autosync set to pullonly.

Keep in mind that Fossil is a DVCS: the syncing and cloning relationships between repos are a graph, with no single repo being a "server" other than by convention. Fossil gives you a lot of control over how artifacts flow along the edges of that graph. These autosync settings are just one part of it. The username and password settings are also part of it.

I didn't express my question well...

I intend to have the following....

Fossil Public Repo --> Clone1 --> clone 2

I was assuming that if I am working on clone2 and I do fossil sync (or auto sync does it), that it will be synchronized with clone1...not with the public repo. Yes or no? Or is the cloning of clone#2, connected somehow directly to the public repo?

I would want that interaction to be full sync in both directions between clone1 and clone2

I want the interaction during sync between clone1 and the public repo to be pull-only from clone1.

I just want to make sure that if set the auto sync in clone1 to pull only, that it will not interfere with full auto sync between clone1 and clone2, presuming I don't also set clone2 to pull only.

if I am working on clone2 and I do fossil sync (or auto sync does it), that it will be synchronized with clone1...not with the public repo. Yes or no?

Yes, autosync won't propagate up the whole chain: it's one step at a time, only. You'd have to do a "push" on Clone 1 to attempt to send those changes to fossil-scm.org, and it'd refuse unless Clone 1 is sync'd with a valid developer user name and password.

is the cloning of clone#2, connected somehow directly to the public repo?

Nope. Clone 2 doesn't even know fossil-scm.org/fossil exists. Only Clone 1 knows that.

I would want that interaction to be full sync in both directions between clone1 and clone2

It doesn't do that by default. You'd have to set up some kind of automation to achieve it. One common method is to run fossil all sync from a crontab file on the NAS. Do it every hour or so, and in most cases it'll be as good as continuous bidirectional syncs up and down the whole chain.

I want the interaction during sync between clone1 and the public repo to be pull-only from clone1.

Then it's not a "full sync", is it?

To get this different behavior, say fossil set autosync pullonly on Clone 1.

A full sync would be leaving this setting at its default, which is on.

But without login creds with developer checkin privileges on fossil-scm.org/fossil, it's almost a distinction without a difference. The only difference is the amount of noise warnings you get on sync of Clone 1.

understood. thanks.

I clone the repo, create a workspace and edit that file and commit it to my own repo..

I wasn't suggesting local checkins at all. For something simple like this, you could just edit the local file then fossil up every time you want to upgrade. Chances are near zero that someone will touch Fossil's source code near this patch unless they're also trying to solve the same problem your local fix does. And at that point, you can just say fossil revert to accept the upstream version.

If you have more complex local changes, then do as drh suggests: maintain a local branch that you merge upstream changes into occasionally.

what would my procedure need to be on the day they release fossil 2.9 and how do I make sure i'm working on a branch for now that is based on the 2.8 release of fossil, ignoring changes after that?

Today:

  $ fossil up release
  ...hack on src/diffcmd.c...
  ...build and test...
  $ fossil ci --branch my-local-changes
  ...hack and test some more...
  $ fossil ci              # no --branch needed now

That gets you a branch with its base at the 2.8 release point.

Then when 2.9 comes out:

  $ fossil up              # or fossil pull
  $ fossil merge release
  ...build and test...
  $ fossil ci

I recommend that you keep at least three checkouts: your branch's checkout (created above), the latest release, and trunk. That lets you compare code among them without continually switching between branches/tags in the single checkout directory. I use the directory scheme ~/src/fossil/branch-or-tag myself.

Maintaining the release checkout is tricky because drh chooses to use "release" as a tag in the fossil-scm.org/fossil repo, not a branch name: saying fossil up will fast-forward a release checkout to the tip of trunk. You have to remember to say fossil up release in that directory each time to keep only release versions in that checkout directory at all times. On the one hand, you only have to give that command once every 3 months or so, but on the other, you have to remember to add the release bit on a command you only run once every 3 months or so. :)

This is why you don't want to say fossil up release in the second group of commands above: as long as you give the command without arguments, you stay on the my-local-changes branch.

I don't entirely understand this bit you said...

Maintaining the release checkout is tricky because drh chooses to use "release" as a tag in the fossil-scm.org/fossil repo, not a branch name: saying fossil up will fast-forward a release checkout to the tip of trunk. You have to remember to say fossil up release in that directory each time to keep only release versions in that checkout directory at all times. On the one hand, you only have to give that command once every 3 months or so, but on the other, you have to remember to add the release bit on a command you only run once every 3 months or so. :)

Can you elaborate a bit more about fossil update...and nuance there? I think for now I'm just going to base my hack branch off everything else up to date..the tip of the trunk. Occasionally i will merge, build and test the latest one. Sounds pretty reasonable. If I need to create something off the version release, I'll cross that bridge when I get there.

But generally I just want to understand the update command a bit more. I more or less thought that if you say update with the name of a branch, then it makes that workspace basically the branch in question..not sure what it does to uncommitted and changes files in the workspace in that case. And right now its not clear to me what update without a version or branch value is going to do. I am going to read more bout this...

I more or less thought that if you say update with the name of a branch, then it makes that workspace basically the branch in question..

That's the problem: "release" is a tag in the fossil-scm.org/fossil repo, not a branch. Therefore:

  $ fossil up release
  $ fossil up

...will send you to the tip of trunk, always, because the "release" tag is on trunk, and the second command means "send me to the tip of the current branch."

This is why I advocate making "release" a branch with releases merged into it, rather than a reused tag. You can see that working pattern here, for example.

Under both schemes, fossil up release takes you to the latest release, but under my scheme, following that with fossil up has no effect until another merge is made into the release branch. You have to explicitly say fossil up trunk to get off of the release branch and onto trunk.

Thought experiment: Suppose I were to go back in the SQLite repo and add a new "release" branch, and merge each release into that branch, as suggested above.

On benefit of this approach is that Git mirrors work better. Git has a limitation that a tag (like "release") can only be applied to a single check-in. So in the Git mirror of the SQLite repository (seen at https://github.com/sqlite/sqlite) on the most recent release is actually tagged with "release".

So, I thought I might try going back and constructing a release branch for SQLite, making generous use of the --date-override option to "fossil commit" so that the new check-ins on the "release" branch occur at the correct times.

But, it turns out the releases of SQLite are not linear. Some branching would be required. Referring to the list of all tagged releases at https://www.sqlite.org/src/timeline?n=1000&t=release (I only started tagging the releases about 10 years ago, so older releases do not appear on that list) you can see that version 3.19.4 occurred after 3.20.0. Similarly versions 3.18.1 and 3.18.2 occurred after 3.19.0.

How do you handle out-of-sequence patch releases in the release-as-branch workflow?

Further question: Does Git allow you to have both a tag and a branch by the same name? And if so, what does it do if you type "git checkout release" and there is both a tag and a branch named "release"? (When Fossil encounters that situation, it picks whichever has the most recent timestamp.)

On benefit of this approach is that Git mirrors work better.

Yes, and it also means someone can have Fossil enforce their intention to only use release versions. Users wanting that will comprise a significant fraction of any large software project's user base.

I thought I might try going back and constructing a release branch for SQLite, making generous use of the --date-override option to "fossil commit" so that the new check-ins on the "release" branch occur at the correct times.

I've done similar things a few times myself. In my experience, the naive approach gets you a long way towards your goal. Then dealing with exception cases involves either a bit of manual rework or an adjustment to the conversion script followed by a re-run on a fresh copy of the repository, iterating until you get a repository that you're willing to push out to the public.

The script is just a loop over a list of checkin IDs by tag — easily extracted from SQL or fossil commands — with the inner part of the loop building and issuing a few commands to create each merge, sanity-check it, and check it in.

The sanity checks are easy to code with commands like test -z "$(fossil changes --conflict)". I wish there was a mode to get a zero/nonzero exit status instead of checking for output size like that, though. A --quiet flag, perhaps.

it turns out the releases of SQLite are not linear.

I think I'd handle that by having the outer loop sort the checkin IDs by version number first, then by commit date, then have it skip an iteration if the current checkin's date is older than the one handled by the prior iteration. Then you just go back in and manually merge the few exceptions.

Once all of this is done, you might want something like my mkrel script to maintain this release branch with a fixed process.

I only started tagging the releases about 10 years ago

You could extract checkin IDs from the comment stream or the ChangeLog and build the first part of the release branch from it.

How do you handle out-of-sequence patch releases in the release-as-branch workflow?

Part of what you're saying by having a release branch, singular, is that there is always "the current release," singular. If there are exceptions to that, I think you'd always want to handle them manually.

I just want to say that fossil rocks and the support from you guys is terrific.

I do not plan to do extensive tweaks to the fossil source. that is a rabbit hole I'd rather avoid and just try to use fossil as it is. So it sounds like I probably don't need to create a branch for this per say.

I was reading somewhere on the fossil site that the trunk of fossil source is considered to be highly reliable...so that I don't need to necessarily confine myself to released snapshots...I could basically update from the public repo at any time, such as right now...I'd get v2.9 in its current developing form...but could feel confident that this developing version of 2.9 would be stable and no problem to work with. Is that still the consensus?

I update to trunk anywhere between hourly and quarterly depending on the union of what's going on with Fossil and what's going on at the site I'm updating my Fossil checkout from:

• If things are calm both places, I do it whenever I get around to it, and it's almost always fine.

• If things are going wild on Fossil's trunk at the moment I update, I do so only when I choose to participate in that particular kerfuffle. Not every new feature of Fossil interests me.

• If things are wild locally, I leave my Fossil binaries the heck alone; I've got other things to worry about.

There's probably only been about 3 times in the ~5 years I've been using Fossil where something has "broken" on trunk in a way that's prevented me from getting something important done, and in every case that comes to mind, it's because I was trying to make use of some brand-new feature. So, the paths out of the trap are obvious: either stop trying to use the feature until it stabilizes or wait for a fix and try to move forward again.

I don't recall ever having to roll back to an old binary in order to move forward.

Keep in mind also that "release" in Fossil-land generally means "Things are calm now, and it's been several months since the last release, so let's tag it and move forward again." Fossil trunk is calm before the tag, and calm after the tag; it's still trunk.

Please rebuild using https://www.fossil-scm.org/fossil/info/a072be1eafe487c6 or later and let me know if the changes in that check-in resolve the issue.

I just tested it and seems to work. Makes sense to use the hash in the repo version of the temporary file name.

I don't know if you noticed the other comment from dab about the --brief option being a little bit broken too, I don't think its related to this change, but if you use --brief and if you also have a third party diff command configured in settings, then --brief doesn't work, it goes to the 3rd party diff instead. Might be worth patching that now too while you're in there...

of course this causes a merge conflict for me at my end, so now I get to learn about how to resolve merge conflicts. Does fossil have any nice way to help me do that? I see that when I did the merge commend of the this new change, into my branch, it gave me a merge conflict, So I immediately did an undo, until I can understand what I'm doing...

I presume that if I do a merge, then I get a merge conflict...there are now three extra files sitting there for each merge conflict...base, original and merge. I'm assuming that I can do a 3 way merge operation using those, resolve the conflicts in the merge tool of my choice and then commit the change to the actual .c file.

Is that correct?

I notice that fossil has the ability to set the gmerge-command setting to call a tool to do that for me, but I can't figure out which fossil command In use to actually launch that?

Well figured out that fossil will launch the gmerge-command during a merge operation that has conflicts.

One thing i wish could be better is that when I save the merged file and quit the visual merge tool, I would like my saved version of the file to not be overwritten. Right now the merge command upon realizing there is a merge conflict, saves the file with a bunch of merge conflict messages inside it (AFTER THE VISUAL TOOL QUITS), even after I have resolved them in a visual merge tool and saved them to the file by that name. I have to make sure to save the merge resolution to some other file name and then after its all finished, go find them and manually move the resolved files to the correct filenames as desired. Ick..

Also when I try to use "%output" as the final argument for the gmerge-command, fossil is coughing up an error that the diffcmd.c-output file is not valid... So I had to remove it...and simply manually specify a file name to save the output to each time...which can't be diffcmd.c in this case, it had to be some other file name like diffcmd.c-output, and then after fossil merge is done, go back and manually move diffcmd.c-output to diffcmd.c

The above could be made easier... unless there is some workflow I'm not aware of.

Looking at https://fossil-scm.org/fossil/doc/trunk/www/checkin_names.wiki, I see special tags for referencing the current checkup. What I don't know (and not able to try, right now) is if these tags will work for the web interface when "fossil server" is run inside a checkout.

Also, there is "ckout" tag, but the documentation implies it only works for "embedded documents".

It would be very handy if those special tags would work in the web interface.

However, one question I have is that fossil appears to put "~0" onto the end of the repo version of the file. This has the unfortunate consequence of disabling Vim's syntax highlighting since it doesn't see the filename extension anymore. Is there any way around that, anyone can think of?

Less than ideal (and likely something you are aware of), but if you are using Vim you may add a modeline to the files you are manipulating. Or, if you are using always the same file types (say, C files), you may set vim -d -c 'set ft=c' as your gdiff program.

Are there any other command line terminal diff programs that people are using with fossil successfully that are colorized and more intuitive then diff and sdiff?

Sometimes I use:

fossil diff | cdiff

See also colorized fossil diff in the terminal in the old mailing list.

For side-by-side diffs, other than Vim I only know of icdiff.

I’m also using visual studio code for visual diff and it does syntax highlighting also if the file extension is intact. It’s quite nice actually. The above hack solves the problem but I will need to setup a proper clone of the fossil repo in order to really manage maintaining this “fork” running forward. It’s a trivial code change but maintaining a fossil fork and remembering to do that is not.

For the terminal diff I couldn’t get cdiff to work. It seems dependent on git. I didn’t try to find icdiff. Most people seem to be using vimdiff for terminal diff and merge.

icdiff works well, thanks for recommending it. for actually merging stuff, I guess vimdiff is the one to use. I'd probably rather just handle merges on my mac with one of many different merging tools...

for actually merging stuff, I guess vimdiff is the one to use.

Vim rocks, IMO, although it has a learning curve. For conflicts, I use this setting:

fossil settings gmerge-command 'vim -d "%baseline" "%original" "%merge" "%output"'

When a conflict happens, for each conflicting file Fossil automatically opens Vim with four (!) windows opened side by side, corresponding to the arguments above. The rightmost window (-output) is the one for the fixed file and is initially empty. The setup may be puzzling at first, but then you may proceed as follows: copy the content of one of the first three windows into the fourth window (you may just type 4dp for that). You may then want to remove the window you have copied from with :diffoff (and optionally close such window). After that, :diffget and :diffput (plus the Normal mappings do and dp) are your friends. When you have fixed your file, save the output buffer and exit Vim.

If a four-way diff confuses you (or your screen is not so wide), you may leave the baseline out:

fossil settings gmerge-command 'vim -d "%original" "%merge" "%output"'

In Vim, copy -original to -output then close -original (:diffoff then :bd) and now you have a standard two-way diff between -merge and -output.

For normal diffs (not conflicts), I just set fossil settings gdiff vimdiff.

I also have these mappings to jump between conflict markers:

nnoremap <silent> ]n /\v^[<\|=>]{7}<cr>
nnoremap <silent> [n ?\v^[<\|=>]{7}<cr>

Finally, Vim can use one of several diff algorithms (see :help diffopt). For example, set diffopt=algorithm:patience may produce better diffs.

Hope this helps!

As an alternative to maintaining a code patch in Fossil, you could wrap your diff command in a script.

For example:

#!/usr/bin/perl
# Filter input arguments for diff program.

my $diffcmd = 'vim -d options file0 file1'; # 'options', 'file0' and 'file1' will
                                                # be replaced with options or filenames

my $usage =<<_EOD_
Usage: $0 [options] file1 file2
_EOD_

my $options = '';
if ($ARGV[0] =~ /^\s*-/)
{
	...; # process options (unless very simple, use Getopt::Std or Getopt::Long)
}

if (@ARGV != 2)
{
	die $usage;
}

for (@ARGV)
{
	if (/\.([^.~])~\d*$/)
	{
		$_ .= ".$1";
	}
}

$diffcmd =~ s/options/$options/;
$diffcmd =~ s/file1/$ARGV[0]/;
$diffcmd =~ s/file2/$ARGV[1]/;

system($diffcmd);

nice idea!

nice catch