Fossil Forum

Removing Credentials from History from within a file
Login

Removing Credentials from History from within a file

(1.2) By xerusf on 2023-11-20 10:17:43 edited from 1.1 [source]

I read up on all threads and articles I could find on purging and shunning (not that many ^^), but could not find a solution to this simple common problem the git community has long solved:

How do I delete accidentally committed credentials from a file throughout history without completely removing the containing file?

Conclusion

First of all, changing the credentials is indeed much more sensible than hunting down the artifacts.

I cannot remove the content from within the files, but I can make the versions containing the credentials vanish from history, though unfortunately in the check-ins inbetween the whole file will then be missing, meaning those cannot really be used anymore. When shunning an artifact, make sure to use a 64-character artifact id as obtained by feeding a shorter one into fossil whatis or finding the file in fossil artifact COMMIT-ID, as fossil info seems to prefer spitting out 40-character hashes which the /shun interface accepts, but does not actually resolve!

(2) By Stephan Beal (stephan) on 2023-11-18 13:43:21 in reply to 1.0 [link] [source]

How do I delete accidentally committed credentials from a file throughout history without completely removing the containing file?

In fossil, you don't. Fossil doesn't track any individual unit smaller than a file.

(5) By xerusf on 2023-11-18 19:36:03 in reply to 2 [link] [source]

Okay, so here is what I tried: I shunned every version of the file containing the credentials (gladly my history is of manageable size) and rebuilt the repository.

Yet somehow nothing changed when I look at the diffs, and the file still appears intact when I run fossil artifact SHUNNED-HASH. I am baffled how this is supposed to work.

(6) By Warren Young (wyoung) on 2023-11-18 20:16:45 in reply to 5 [link] [source]

Either you're putting commit IDs in where it is expecting artifact IDs or you are neglecting to consider the fact that the shun list doesn't sync unless you force it. Shunning on a central server doesn't affect your local clone until you say "fossil conf pull shun".

(8.1) By xerusf on 2023-11-19 08:09:32 edited from 8.0 in reply to 6 [link] [source]

I am shunning on my local clone to test it first, and I put in the artifact IDs. As I said the exact artifact ID I shunned I can still view via CLI.

Do I need to run something in my local checkout as well?

Look here:

❯ fossil config export shun -
# The "shun" configuration exported from
# repository "/home/janek/data/1-projects/ftt/cybercluster/solidcluster.fossil"
# on 2023-11-19 08:08:28
config /shun 87
1700306228 '659a39ddf7cd25aacaafccfbe91a5ceec0a051beab20c85da512a3b499dc0f4a' scom NULL
config /shun 87
1700306228 '8c9d70deafb4c021e869213f04b7c6095f59da98dd5148e9e5c88dd6aae6713e' scom NULL
config /shun 63
1700335310 'dde679f10e452ed52c94de2be3e5d51d913bd9cb' scom NULL
config /shun 63
1700335310 'b007fae5626770c1cb4a0317119709bca01f57e6' scom NULL

❯ fossil rebuild
  100.0% complete...

❯ fossil artifact dde679f10e452ed52c94de2be3e5d51d913bd9cb
require_relative "boot"

require "rails/all"

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)
...

(9) By Stephan Beal (stephan) on 2023-11-19 14:29:04 in reply to 8.1 [link] [source]

Do I need to run something in my local checkout as well?

IIRC, shunning does not actually clean up the shunned artifacts until the next time "rebuild" is run.

(10) By spindrift on 2023-11-19 15:45:11 in reply to 9 [link] [source]

While this is true, a rebuild is explicitly being run here.

(15) By Preben Guldberg (preben) on 2023-11-20 08:05:29 in reply to 10 [link] [source]

In the example, you seem to have shunned two 64 byte and two 40 byte artifact IDS.

Did you by chance try to shun dde679f10e452ed52c94de2be3e5d51d913bd9cb using an incomplete 64 byte ID?

(17) By xerusf on 2023-11-20 09:44:59 in reply to 15 [link] [source]

The ID is recognized in the web interface. Note that the artifact is included in multiple commits, but that shouldn't concern fossil, right?

Here is how I obtained it:

❯ fossil finfo config/application.rb
History for config/application.rb
2023-11-18 [a10fcde1a0] fix: move mailer config to rails credentials (user: xeruf, artifact: [596857f4aa], branch: trunk)
2023-11-18 [6a3230eb71] chore: securely check-in credentials (user: xeruf, artifact: [aa74dc4ab6], branch: trunk)
2022-10-16 [8f2868bbad] fix: check in dev tooling (user: xeruf, artifact: [b007fae562], branch: trunk)
2022-10-16 [ddab5f61b0] feat: improve styling and translations (user: xeruf, artifact: [dde679f10e], branch: trunk)
2022-05-16 [482e8f8fa6] Integrate slider with html form (user: xeruf, artifact: [f3d364538d], branch: trunk)
2022-05-11 [26cf405f1a] config: add host (user: cat, artifact: [88f34ac290], branch: trunk)
2022-05-11 [ef2f1547bc] Add config (user: xeruf, artifact: [70137010f0], branch: trunk)
❯ fossil info dde679f10e
hash:         dde679f10e452ed52c94de2be3e5d51d913bd9cb 

(19) By Warren Young (wyoung) on 2023-11-20 09:57:00 in reply to 17 [link] [source]

And what happens when you say "fossil artifact dde679f10e452ed52c94de2be3e5d51d913bd9cb" after the rebuild?

(20) By xerusf on 2023-11-20 10:11:41 in reply to 19 [link] [source]

See (8).

But I found the issue, see my edit of (1), maybe the info command can be adjusted or the shun menu changed to only accept 64-char hashes or just resolve them itself, as this was a lot of unnecessary pain.

(21) By Warren Young (wyoung) on 2023-11-20 11:02:29 in reply to 20 [link] [source]

The shun page accepts only full hashes to avoid ambiguity, but Fossil supports two different hash types, differing by and recognizable by length.

Like Git, Fossil started off supporting 160-bit SHA-1 hashes, which print out in hex as 40 characters. 64-character hashes are 256-bit SHA-3 hashes, which we've been using by default for many years now, since shortly after the first major SHA-1 hash weaknesses were published.

Are you reporting that Fossil can't shun SHA-1 hashes any more?

(22) By Preben Guldberg (preben) on 2023-11-20 17:25:01 in reply to 21 [link] [source]

Are you reporting that Fossil can't shun SHA-1 hashes any more?

I think the complaint is more that an abbreviated 40 byte hash obtained from fossil output was accepted for shunning, when the intended target was the real 64 byte hash.

I can see that being surprising if you are copy and pasting valid hashes.

In shun-list-review branch I added a "Review" button next to "Shun" to let you check the hashes.

Would that be useful?

(11) By spindrift on 2023-11-19 15:50:59 in reply to 8.1 [link] [source]

Have you checked on the local clone that the shunned is recognised? Perhaps some weird database write error is being problematic.

fossil rebuild
fossil ui

Then navigate to

/shun
in the web ui and see what is there.

The complete shunning list for a repository can be viewed by a user with "admin" privilege on the "/shun" URL of the web interface to Fossil. That URL is accessible under the "Admin" button on the default menu bar. Items can be added to or removed from the shunning list. "Sync" operations are inhibited as soon as the artifact is added to the shunning list, but the content of the artifact is not actually removed from the repository until the next time the repository is rebuilt.

(12) By Stephan Beal (stephan) on 2023-11-19 16:17:18 in reply to 11 [link] [source]

fossil ui ... Then navigate to ... /shun

Tip:

fossil ui -page shun

(13) By spindrift on 2023-11-19 20:45:45 in reply to 12 [link] [source]

That's handy, thank you.

(18) By xerusf on 2023-11-20 09:45:50 in reply to 11 [link] [source]

yes, I added the entries via the UI and they are still there

(14) By Warren Young (wyoung) on 2023-11-20 04:41:45 in reply to 8.1 [link] [source]

It works perfectly in my test here:


$ cd ~/tmp
$ mkdir x
$ cd x
$ fossil init ../x.fossil
$ fossil open ../x.fossil 
$ echo hi > README.md       
$ fossil add README.md
$ fossil ci -m 'added readme'    
New_Version: cfc9ef5811d6abb29e91db677f47bba148e3e0a29042ca15ee0e45034542fc63
$ fossil art cfc9ef5811d6a
C added\sreadme
D 2023-11-20T04:34:33.911
F README.md a57d4ad9fb250f5a211a3ed89d378cd5aaa6fc8da2d626cac4868d9964840965
…
$ fossil art a57d4ad9fb250f5
hi
$ fossil ui -page shun
Listening for HTTP requests on TCP port 8080

At this point I then pasted the full file artifact ID (a57d4ad9fb250f5…) into the box, saved it, and hit the Rebuild button on that same page, then Ctrl-C'd out of fossil ui and proceeded thus:


$ fossil art a57d4ad9fb250f5
cannot resolve name: a57d4ad9fb250f5a211a3ed89d378cd5aaa6fc8da2d626cac4868d9964840965

The file is gone, as I told it to, but the commit artifact mentioning that shunned file (cfc9ef5811d6a…) remains.

(3) By Andy Bradford (andybradford) on 2023-11-18 15:03:21 in reply to 1.0 [link] [source]

> but could  not find a solution  to this simple common  problem the git
> community has long solved:

How does one go about deleting  lines of files from historical copies of
the file already committed  in Git? This is news to me  that it has such
capability.

Andy

(4) By Daniel Dumitriu (danield) on 2023-11-18 15:16:08 in reply to 3 [link] [source]

Don't ask: for each commit, apply some filters to some files; replay commit. And it's not git, it's a python script. Fully gitty, I'd say.

(16.1) By Konstantin Khomutov (kostix) on 2023-11-20 09:24:23 edited from 16.0 in reply to 3 [link] [source]

Git provides git filter-branch which basically works by invoking a provided shell script (or any other program) on each commit of the branch passed to the tool. It has multiple modes of operation (say, completely deleting a file can be run substantially faster than actually patching it and creating a new commit from the result).

The resulting contents of the affected branches will obviously have different commits from those before the operation.

There exist a bunch of 3rd-party programs building upon this basic tool; @danield referred to one of them.

(7) By sean (jungleboogie) on 2023-11-18 20:30:47 in reply to 1.0 [link] [source]

How do I delete accidentally committed credentials from a file throughout history without completely removing the containing file?

Wouldn't it be better to change the credentials?

In the future, you can use fossil diff to see what has changed before you commit.