Hi,

Does fossil support any kind of string replacement? I'm remembering rcs and $Id$ which would get replaced at checkin by:

$Id: filename revisionnumber date time author state locker $

(with appropriate actual values for filename, revisionnumber, etc.) and would keep it up to date on subsequent checkins. I've checked into this with git, but never fossil.

The advantages are obvious, but I don't know of disadvantages.

Will

Does fossil support any kind of string replacement?

No.

I don't know of disadvantages.

If it did, what it actually checked in would differ from what you told it to check in because it would necessarily have to modify files between the time you said "check this in" and the time it told the date base "put this in the database."

Without wanting to put words in anyones' mouths: based on prior discussions of this topic, most or all of fossil's developers are fully convinced that fossil should only store exactly what you tell it to and that the user should test what they check in before doing so. If fossil modifies files, it means it's literally impossible to test a file before checking it in, as what gets checked in necessarily was not what was tested.

The main disadvantage is that the $Foo$ strings might appear in binary content, which would likely be wrecked on checkout.

This is why Subversion made you set it on a per-file basis with its svn propset svn:keywords command. (See the warnings at the end of that document for the "why" of this not being the default in Subversion, as it was in CVS and RCS.)

Chasing this solution feels like an XY Problem. Instead of telling us about the problem with your intended solution, tell us what problem you intend it to solve. Automated version number building in About screens and --help messages, perhaps?

A hook cold be set up to Review the files automated just before checked in, but it is just a hook and the work remains on you

but I don't know of disadvantages.

Another one is that you cannot inject a version number into a file without changing that file's content, and changing that content changes the manifest, which changes the version (a hash of that manifest). The version cannot be calculated until all content for the checkin is known, at which point even a single bit of change would invalidate the version hash.

Alright already :). Sheesh. I'm not trying to start a debate on the merits. The cause has apparently already been lost and I'm not equipped to fight it in arrears.

I just miss being able to put a $ID$ tag in my mom's apple pie recipe so that I know what version it is when I read it. Sometimes, I like the old version and sometimes the new... and I'm too lazy to keep track myself by adding text representing the version... not really, but hopefully you get the idea, which is not fringe, regardless of where modern scm tools have gone with hashing - CVS is still used for plenty of currently maintained projects and the tag replacements made in the source are still helpful in 2024.

Y'alls arguments make very fine sense, don't get me wrong. I was just hoping... I have never figured out how to write hooks, maybe this is my motivator.

Will

Here is how I am using the before-commit hook to check the files that are going to be committed before they are ... a proper return code could avoid the commit or permits it ... the files might be modified (string replacements as you like).

D:\project>fossil.exe hook list
  0: type = before-commit
     command = D:/UTL/lua53.exe D:/REPO/fosbuild.lua before-commit %F %A 0
     sequence = 10

  1: type = after-receive
     command = D:/UTL/lua53.exe D:/REPO/fosbuild.lua after-receive %F %A 0 --
     sequence = 10

The Lua script itself looks like this (it is a working skeleton for more useful work than just listing):

print("Time", os.date("%Y-%m-%d %H:%M:%S") )
print( "Program", arg and arg[-1] )     -- first parameter (not counted)
print( "Script",  arg and arg[0] )      -- 2nd parameter (not counted, too)

local sqlite3 = require("lsqlite3x53")
if sqlite3 then
  db = sqlite3.open_memory()   -- sqlite3.open("database.sqb")
  print( "lsqlite", sqlite3.version() )
  db:exec([[select syslog(1, "]]..(arg and arg[1] or "?")..[[ called from inside of lua53.exe fosbuild.lua")]])
end

local checkoutFolder = "./"

local rc = 0
for n,item in ipairs( arg or {} ) do  -- print all counted parameters
  if db then
    local msg = string.format([[select syslog(0, "n= %d, item= %s")]], n, item )
    db:exec( msg )
  else
    print( n, item )
  end
end
if arg and arg[#arg]=="--" then         -- check if last parameter indicates 'stdin used'
  for line in io.lines() do         -- read all lines from stdin
    if db then
      local msg = string.format([[select syslog(0, "stdin= %s")]], line )
      db:exec( msg )
    else
      print( line )
    end
  end
  rc = tonumber(arg[#arg-1]) or 0   -- use last parameter as return value
else
  rc = tonumber(arg and arg[#arg]) or 0     -- use last parameter as return value (no stdin usage)
end

local function processFile( fname )
  local fh = io.open( fname, "r+" )
  if fh then
    local firstLine = fh:read("L")
    if firstLine then
      print( fname,   "line 1st=", firstLine )
      local pos,msg = fh:seek("end")
      if pos then
        print( fname, "seek end=", pos )
      --fh:write( firstLine )           -- for now avoid modifying the file
      else
        print( msg )
      end
    end
    fh:close()
  end
end

local function listFile(id,fname, optProcess )
  local rv = nil
  if id == "before-commit" then
    for line in io.lines(fname) do
      local key, value = line:match("(.-)%s+(.+)")
      if key == "tags" then
        rv = value
      end
      if db then
        local msg = string.format([[select syslog(0, "%s= %s")]], fname, line )
        db:exec( msg )
      else
        print( line )
      end
      if (key=="checkout") then
        checkoutFolder = value
        print( "checkoutFolder =", value )
      elseif (key=="EDITED" or key=="ADDED") and type(optProcess)=="function" then
        optProcess( checkoutFolder..value )
      elseif (key=="UNCHANGED") then
        print( "nothing to do for", value )
      end
    end
  end
  return rv
end

local branch = arg and listFile( arg[1], arg[3], processFile )
--[[
if branch then
  local fh = io.open( branch .. ".comment", "a+" )  -- append update mode on comments-file
  if fh then
    fh:write("\nprocessed by hook type before-commit\n")
    fh:close()
  end
end
--]]

if db then
  local msg = string.format([[select syslog(1, "return %d")]], rc )
  db:exec( msg )
else
  print( "return",  rc )
end

_ = db and db:close()

return rc

Do good and not evil with this code, it is free for anybody and any usage.

Just a small FYI, but i think this script will miss the case where files are renamed and edited. ie,

EDITED old-filename -> new-filename

There shouldn't be any harm though since processFile checks for the file's existence.

I don't know if there's other missing cases, but i'm particularly sensitive to this one after some recent trauma..

Thank you very much for this hint. - I seldom do renaming, that must be the reason why I missed that case ... I will do some more tests for completeness.

Any more feedback is appreciated.

Hi Will,

I'm not sure if you missed Ta's reply mentioning Fossil's manifest setting: this causes Fossil to write one or more files (see Ta’s post), which can be referenced by other files in your repo.

Of course, that only works if the file format you're using supports including stuff from a text file.

In my code, I routinely include the manifest file(s) as resources, which my application then reads at runtime.

Martijn

Ta,

This sounds interesting. So, what's a manifest rtu?

Will

I saw it, but it was way over my head. I've asked for clarification :).

This.

Got it. Thanks. That's handy.

I did the following: mkdir .fossil-settings/ vi .fossil-settings/manifest rtu

then made some changes to a file, added the changes, and checked in. Like magic, manifest, manifest.uuid, and manifest.tags showed up.

How can I check the hashes against reality - in manifest it has a checksum, but It's not sha1, 256, 512, etc. From reading it seems like it's a stronger version of sha1, and y'all call it sha3... how can I generate it from the source file, say vi.txt.

sha3sum vi.txt isn't an option, so what is?

Also is the checkin hash a hash created from all of the changes made to the repo, or is it generated as a unique id with no attachment to what's in the repo?

-will

y'all call it sha3

Yes. So does NIST.

how can I generate it from the source file

Ask Fossil.

is the checkin hash a hash created from all of the changes made to the repo, or is it generated as a unique id with no attachment to what's in the repo?

Neither.

See also…

...to put a $ID$ tag in my mom's apple pie recipe so that I know what version it is when I read it.

Not sure in which document form you store the recipes. In case it's just some plain text or other format that Fossil UI can render, then the UI's File view (or rather Artifact view) will show the details for the version, date, commit info etc. This page may as well be printed, so all that info is retained.

Alternatively, you may consider "building" your recipe library (using make of some other script), so that the source files get "stamped" in some destination dir, maybe also converted into PDF or whatever target format.

This separates "source" form (kept in VCS) and "target" form (which could be rendered and deployed as desired).

Thanks for the great links! It's been so long since I used ssl for digests, but that's an option too:

openssl dgst -sha3-256 vi.txt

Off to read that File Formats page... good info, but not light reading :)

I think I'll prolly just write a python script to do replacements. If it works out, I can hook it up to fossil. I'll call it markver, have it keep simple version numbers in a registry that aren't based on anything other than an explicit $VER$ tag or incrementing from 0. Have it be dot based, semantic versioning.

say apple-pie.txt contains nothing but recipe. markver adds it to the registry w/version 0. next checkin, the registry gets 0.1, 0.2, etc. I wanna bump the version, I edit/add $VER +1.0$, registry gets updated to 1.2. now if I make changes and add $Id$ it can run pre-checkin, get replaced with $Id ver 1.3, wsenn, halleluja!$ and not mess with hash recursion or any nonsense like that. If I never mess with explicitly changing it, at least it'll be incremented and reasonable 0,1,2...

I'm sure It'll be fancier, don't wanna have collisions, but I've done more complicated work with mp3 renaming with undo. Now, if I could just find the time :)

When I think about it, hash recursion's a pain, but I can prolly hash the contents outside of the replacement and store it in the replacement without incident, too.

$id v1.3.1, niftyhash, wsenn, yippee$

Then my id message is correct, useful, and not conflicting.

hash recursion's a pain,

It sounds like you're trying to recreate Fossil's method of hashing artifacts from outside Fossil, but I can't see why you wouldn't just ask Fossil for the info you need:

$ fossil finfo -i apple-pie.txt
artifact:   1f55e8f5951bc33ce964a096ad7a9ea4db4f08d5d8acbd505401750cc3f47e23
size:       NNN bytes
mtime:      2024-07-31 18:45:12
file:       apple-pie.txt
            part of [3311d631d4] by wsenn on 2024-07-31 18:45:12
            Last commit comment appears here

Substitute as much of that into the file before printing as you like.

The "part of [$COMMIT_ID]" line gives the most reliable info for looking up the exact version referenced, but you can also do it by any unique prefix of the artifact ID.

Don't like hashes? Fine. You can also reliably look up historical versions of files given only the timestamp and file name, which that report also gives you.

The problem to solve is so typical in corporate world (especially large enterprises with many teams) that it's why it was by default in RCS - yup, just half a year ago I was working for mobile operators' billing-writing company, and this billing, Ensemble, worked at e.g. Vodafone, and it was using RCS, consisting of thousands of binaries, not counting systems "around" it. The problem is called - debugging from which parts that particular improperly working thing was assembled, in the absence of single point of responsibility. Yes, one may argue that this is organizational problem "then that company should fix that" rather than technical - but many technical tools were created exactly to solve organizational problems, especially in problem domain of source code control. For example, such burning success of Git is largely provided by fact Git is a collaboration tool, not just source files tracking tool. I'd prefer Fossil being more popular so that it can be more often offered as an alternative to Git, but in many cases such offers will be rejected (by those who make decisions) due to lack of collaborative features. And unfortunately for some of them it's position of Fossil developers preventing them to appear. This thread's topic is very small part of it (there are bigger ones), but let's return to it...

So corporations with brain-damaged management? Alas, not only! Just today I had a fight with eBPF on Ubuntu: I've put a flag BPF_F_TIMER_ABS to timer call, and - BOOM! - it doesn't compile! But my program had #include <linux/bpf.h> and example from tools/testing/selftests/bpf/... in kernel sources was using same include. Of course, kernel sources were matching running kernel which was installed from official package. Just... the .h had very different size and been from another package! Which had unknown flags in dpkg output instead of usual ii.

At this point, it would be MUCH helpful to me if the .h contained branch, date and revision from VCS - so I could compare in-place if it is safe to replace file by taking it from kernel sources. Instead of spending hours on screwed up building systems of distro, Linux et. al. Of course I did not do such wasting of time, and just dropped one-line #define to my own file.

When talking about XY problem, one must recall that (material/technical) progress is all about doing same things with less effort/resources, or in other words, doing more things with same resources/effort as before; this is crucial at least as growing population of Earth must eat from same square of agricultural land.

So if perceived "right" solution, instead of line expanding, requires hours of digging, potentially unsolvable at all due to human factor, then this CANNOT be right solution. At all.

Returning to keyword expanding and "intact files". Git has much the same argument as Fossil above, BUT actually THERE IS case where it became not true - line endings. You know, those CR vs CR LF vs LF vs ... things. And nobody in sane mind will say that files must be kept intact as they were committed first time from different OSes, putting other people on other OSes to pain of converting this zoo.

No, instead the right thing is to have a "canonical form" for file, which will be hashed to be kept in DVCS - and converted in/out on checkin/checkout. So the same with keywords. The "problem" Stephan said - about testing - is solved by just having an option to disable conversion and work with raw form. Which will be almost never really needed for things like line endings.

Thus, as Fossil already has SVN-like properties on files - being better than Git here - it would be quite logical to expect having handlers for some of these properties, like keyword expansion, if those is enabled in repo. One of fundamental Unix way principles - "tools, not policy".

I think such a feature would be very useful and maybe we could be more creative in finding a way to achieve it.

How about if we go the other way around - you still put in the database exactly what has been submitted, with the placeholder intact, and then expand the placeholder with the actual check-in data at check-out, and / or by a separate fossil command? Maybe even teach fossil changes to recognize this and ignore that specific difference... well, then we may need to store somewhere a second hash of the file with the placeholder expanded... but why not? Will it really not be worth it?

Anything that creates a mismatch between unchanged files in the checkout and what's actually stored is likely to create more fresh misery than it removes from the world. Witness all the argumentation over Git's autocrlf setting.

Let's try to learn from other VCSes' mistakes, not recreate them.

The mistake there was "auto" part, not filtering hooks itself. I've seen more complex translations with Git hooks than CRLF in production - between UTF-8 and national codepage on different installations of product (corporate Web Site/API in different countries).