How do I delete accidentally committed credentials from a file throughout history without completely removing the containing file?

In fossil, you don't. Fossil doesn't track any individual unit smaller than a file.

Okay, so here is what I tried: I shunned every version of the file containing the credentials (gladly my history is of manageable size) and rebuilt the repository.

Yet somehow nothing changed when I look at the diffs, and the file still appears intact when I run fossil artifact SHUNNED-HASH. I am baffled how this is supposed to work.

Either you're putting commit IDs in where it is expecting artifact IDs or you are neglecting to consider the fact that the shun list doesn't sync unless you force it. Shunning on a central server doesn't affect your local clone until you say "fossil conf pull shun".

I am shunning on my local clone to test it first, and I put in the artifact IDs. As I said the exact artifact ID I shunned I can still view via CLI.

Do I need to run something in my local checkout as well?

Look here:

❯ fossil config export shun -
# The "shun" configuration exported from
# repository "/home/janek/data/1-projects/ftt/cybercluster/solidcluster.fossil"
# on 2023-11-19 08:08:28
config /shun 87
1700306228 '659a39ddf7cd25aacaafccfbe91a5ceec0a051beab20c85da512a3b499dc0f4a' scom NULL
config /shun 87
1700306228 '8c9d70deafb4c021e869213f04b7c6095f59da98dd5148e9e5c88dd6aae6713e' scom NULL
config /shun 63
1700335310 'dde679f10e452ed52c94de2be3e5d51d913bd9cb' scom NULL
config /shun 63
1700335310 'b007fae5626770c1cb4a0317119709bca01f57e6' scom NULL

❯ fossil rebuild
  100.0% complete...

❯ fossil artifact dde679f10e452ed52c94de2be3e5d51d913bd9cb
require_relative "boot"

require "rails/all"

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)
...

Do I need to run something in my local checkout as well?

IIRC, shunning does not actually clean up the shunned artifacts until the next time "rebuild" is run.

While this is true, a rebuild is explicitly being run here.

In the example, you seem to have shunned two 64 byte and two 40 byte artifact IDS.

Did you by chance try to shun dde679f10e452ed52c94de2be3e5d51d913bd9cb using an incomplete 64 byte ID?

The ID is recognized in the web interface. Note that the artifact is included in multiple commits, but that shouldn't concern fossil, right?

Here is how I obtained it:

❯ fossil finfo config/application.rb
History for config/application.rb
2023-11-18 [a10fcde1a0] fix: move mailer config to rails credentials (user: xeruf, artifact: [596857f4aa], branch: trunk)
2023-11-18 [6a3230eb71] chore: securely check-in credentials (user: xeruf, artifact: [aa74dc4ab6], branch: trunk)
2022-10-16 [8f2868bbad] fix: check in dev tooling (user: xeruf, artifact: [b007fae562], branch: trunk)
2022-10-16 [ddab5f61b0] feat: improve styling and translations (user: xeruf, artifact: [dde679f10e], branch: trunk)
2022-05-16 [482e8f8fa6] Integrate slider with html form (user: xeruf, artifact: [f3d364538d], branch: trunk)
2022-05-11 [26cf405f1a] config: add host (user: cat, artifact: [88f34ac290], branch: trunk)
2022-05-11 [ef2f1547bc] Add config (user: xeruf, artifact: [70137010f0], branch: trunk)
❯ fossil info dde679f10e
hash:         dde679f10e452ed52c94de2be3e5d51d913bd9cb 

And what happens when you say "fossil artifact dde679f10e452ed52c94de2be3e5d51d913bd9cb" after the rebuild?

See (8).

But I found the issue, see my edit of (1), maybe the info command can be adjusted or the shun menu changed to only accept 64-char hashes or just resolve them itself, as this was a lot of unnecessary pain.

The shun page accepts only full hashes to avoid ambiguity, but Fossil supports two different hash types, differing by and recognizable by length.

Like Git, Fossil started off supporting 160-bit SHA-1 hashes, which print out in hex as 40 characters. 64-character hashes are 256-bit SHA-3 hashes, which we've been using by default for many years now, since shortly after the first major SHA-1 hash weaknesses were published.

Are you reporting that Fossil can't shun SHA-1 hashes any more?

Are you reporting that Fossil can't shun SHA-1 hashes any more?

I think the complaint is more that an abbreviated 40 byte hash obtained from fossil output was accepted for shunning, when the intended target was the real 64 byte hash.

I can see that being surprising if you are copy and pasting valid hashes.

In shun-list-review branch I added a "Review" button next to "Shun" to let you check the hashes.

Would that be useful?

FWIW, I still think this would be useful to avoid confusion when copy' pasting hashes.

I'm inclined to merge this unless there are objections.

Have you checked on the local clone that the shunned is recognised? Perhaps some weird database write error is being problematic.

fossil rebuild
fossil ui

Then navigate to

/shun

in the web ui and see what is there.

The complete shunning list for a repository can be viewed by a user with "admin" privilege on the "/shun" URL of the web interface to Fossil. That URL is accessible under the "Admin" button on the default menu bar. Items can be added to or removed from the shunning list. "Sync" operations are inhibited as soon as the artifact is added to the shunning list, but the content of the artifact is not actually removed from the repository until the next time the repository is rebuilt.

fossil ui ... Then navigate to ... /shun

Tip:

fossil ui -page shun

That's handy, thank you.

yes, I added the entries via the UI and they are still there

It works perfectly in my test here:

---

$ cd ~/tmp
$ mkdir x
$ cd x
$ fossil init ../x.fossil
$ fossil open ../x.fossil 
$ echo hi > README.md       
$ fossil add README.md
$ fossil ci -m 'added readme'    
New_Version: cfc9ef5811d6abb29e91db677f47bba148e3e0a29042ca15ee0e45034542fc63
$ fossil art cfc9ef5811d6a
C added\sreadme
D 2023-11-20T04:34:33.911
F README.md a57d4ad9fb250f5a211a3ed89d378cd5aaa6fc8da2d626cac4868d9964840965
…
$ fossil art a57d4ad9fb250f5
hi
$ fossil ui -page shun
Listening for HTTP requests on TCP port 8080

---

At this point I then pasted the full file artifact ID (a57d4ad9fb250f5…) into the box, saved it, and hit the Rebuild button on that same page, then Ctrl-C'd out of fossil ui and proceeded thus:

---

$ fossil art a57d4ad9fb250f5
cannot resolve name: a57d4ad9fb250f5a211a3ed89d378cd5aaa6fc8da2d626cac4868d9964840965

---

The file is gone, as I told it to, but the commit artifact mentioning that shunned file (cfc9ef5811d6a…) remains.

Don't ask: for each commit, apply some filters to some files; replay commit. And it's not git, it's a python script. Fully gitty, I'd say.

Git provides git filter-branch which basically works by invoking a provided shell script (or any other program) on each commit of the branch passed to the tool. It has multiple modes of operation (say, completely deleting a file can be run substantially faster than actually patching it and creating a new commit from the result).

The resulting contents of the affected branches will obviously have different commits from those before the operation.

There exist a bunch of 3rd-party programs building upon this basic tool; @danield referred to one of them.

How do I delete accidentally committed credentials from a file throughout history without completely removing the containing file?

Wouldn't it be better to change the credentials?

In the future, you can use fossil diff to see what has changed before you commit.