Hey,

I haven't been able to use Fossil for a while, for reasons, but I still follow its development and I really love it.

I wanted to highlight this quote from Wired's article on the XZ backdoor,

https://www.wired.com/story/jia-tan-xz-backdoor

Because those changes, known as “commits,” are often batched into collections in a process known as “squashing commits,” it’s not always apparent which exact changes were made by Jia Tan. And the difficulty of tracing which of the many versions of a library like libarchive ended up in which software adds yet another layer of obfuscation. “It’s going to be a bit of a mess pulling on this thread and trying to figure out where all these things ended up,” Scott says.

Security is such a huge deal and it's becoming more and more relevant to people in positions of accountability as more hacks happen.

I think Fossil's avoidance of history surgery would have preserved a lot more of the trail of the hacker(s). There's this "automatically squash" feature when you merge so it could conceivably use another branch for that but GitHub prunes a branch after you merge its pull request, so there's no record of the pre-squash commits. Even if GitHub fixes that, developers who use Git rebase all the time, just for clarity, to make it easier on people doing code reviews -- I know I do. Git could adopt Fossil's philosophy of persistence and still support rebasing just by preserving the state before each rebase to keep the true history. But I don't think Git's going to do that.

So, if anyone's pitching Fossil for use by a government project / group, Fossil's permanent code audit trail would be a SUPER important feature to highlight in light of these attacks (who knows how many are still in progress?).

-- Bill

Yes. You use completely new words and a contemporary example to argue the point I've tried (unsuccessfully) for years to make, which is that keeping an complete, immutable audit trail is important. But this idea tends to provoke a lot of drama and ire among many Git users. I'm still not completely sure why that is, but it is what I have observed, for years.

Thank you for making the point again.

Side track: Should news outlines do the same with their news articles, so that when they make corrections, the changes are still always accessible? It seems to me that this ought to be "best practice" among responsible journalists. I'm not suggesting Fossil is necessarily the right platform for a newspaper, but another software system that keeps track of all published articles using SHA3 (or similar) hashes and keeps all revision history accessible to historians seems to me like it ought to be the norm? Am I completely off-base here?

Hi Richard!

Yes, I should have mentioned that you've been harping on this to deaf ears ever since you fist released Fossil.

But that's kind of why I made this post. I think your points are becoming painfully important now. I hope Fossil can get some traction with large organizations because of this. Perhaps teams that require iron clad audit trails would listen...

Side track: Definitely. Wikipedia keeps a change history for each page -- news publications should too. The XZ Wired article says

Updated 4/3/2024 at 12:30 pm ET to note the possibility of Israeli or Iranian involvement.

So it at least notes the change but browseable would be way better.

Git could adopt Fossil's philosophy of persistence and still support rebasing just by preserving the state before each rebase to keep the true history. But I don't think Git's going to do that.

"The problem" with that, i opine, is that source trees in the scale of the Linux kernel would become untenably large if history could not be pruned. The Linux kernel reportedly gets approximately 120 checkins per day. Granted, only a tiny fraction of projects would genuinely suffer logistically if forced to retain all history, but it would be a sizable (haha!) problem for that tiny fraction.

Hi Stephan!

That could be, but as scaling problems have arisen, Linus and friends have tackled them. Pulls started becoming a problem because Git (logically) keeps a separate copy of each version of each file, so they added diff-based "packs" for rapid syncing and lighter-weight storage. I'm sure they could find a solution to larger logical histories if this was really a priority.

The Linux kernel reportedly gets approximately 120 checkins per day.

Sure, but of the zillions of Git-based projects, one is the Linux kernel, and something well under 1% of the rest have anywhere near the diversity of external contributors, which in turn requires a network of "lieutenants" to manage PRs, squashing things along the way.

Everyone thinks they're the next Google and thus need to build a next-level GitOps stack to feed their worldwide Kubernetes clusters, and then they end up running it in a single box on-prem. 🙄 Sheesh.

it would be a sizable (haha!) problem for that tiny fraction.

I have no problem with the tiny fraction of Git users who need its power choosing it. What I have a problem with is that everyone thinks they need to "…witness the power of this fully-operational battle station!"

The whole operation seems to have relied upon taking over the maintainership of the XZ project. Thus abusing the position of trust to introduce vulnerability.

If you read the details uncovered so far, the perp was even disabling fuzz, using .gitignore to hide some parts of the vulnerability, tucking some pieces within test files etc. [1].

Altogether this is not much about Git, as it is more about OSS community almost failing in collective code review approach. Of course, the degree of sophistication played huge role too.

GitHub has a very capable review system. Squished or not, the changes do not go in automatically. Also, someone trusted needs to review and approve. So there were multiple barriers to entry into the main codebase. All carefully orchestrated to be quietly lowered.

This was an attempted attack on the OSS core values: trust and collaboration. Yet the transparency of the code, another OSS core principle, has helped detect it and possibly thwart the intended damage.

The availability of complete check-in history is not a silver bullet that will magically prevent these kinds of attacks. However it does offer two advantages:

1. The inability to squash commits makes it more difficult to hide nefarious code changes.

2. Having the complete check-in history is analogous to having a flight data recorder in that it empowers investigators to better analyze the attack, figure out what went wrong, and devise effective strategies to prevent a recurrence.

GitHub has a very capable review system

Yes, and if you study the psychology that underpinned this incident, you can only come to the conclusion that code review would not have happened.

What we have here is a maintainer that was tired of maintaining xz/liblzma2, took the first credible offer to take over maintenance, and went MIA. Nobody second-guessed this, nobody checked the work, because everyone thought, "Good, someone's continuing to maintain xz now. It would be a shame if it began to bit-rot. Problem solved, we can go back to ignoring it again; move on to the next."

Code review only works if there's a group of people both willing and capable of doing that code review.

But of course, that's the trick. If you don't put the necessary resources into it, you end up in one of several pathologies:

• Maximum review: Everything's bottlenecked through a QA team or project lead or similar, and nothing gets committed to what we shall call "trunk" (to keep this in Fossil terms) without their say-so. This often ends up idling workers as they sit around waiting for their PR to be approved, without which they can't continue, lest a rejection report derail work they built atop the initial PR.

• Passive review: Taking the lesson from that, the project's leadership makes code review asynchronous, letting people continue with their work in parallel with the goal of getting as close to the Mythical Man-Month as possible by reducing communication overheads and blocking points. This buys a new problem: who does this work in a one-person project like xz? It's part-time for a single developer already.

• Debutante review: Taking the lesson from that, someone from the community is appointed to do the work. Maybe we find some foolish sod to do review work for several projects so we don't have a 2:1 talent overhead, one reviewer per developer. If they're competent to the point of being able to second-guess every line of the presumably-brilliant main developer's code, they're more likely to go off and spend their precious free time working on their own projects instead. If they're lesser lights, why is that we're trusting them to audit the code, again?

• No review: The present situation: all the developers competent to write their own code are doing so, and all the rest are doing something else besides second-guessing code they don't even understand properly.

IMO, GitHub's code review feature is best used as a tool for fending off bad PRs, and little else. It lets you tear apart some nasty ball-o-hackage that someone threw at you and explain — line-by-line, if necessary — why you aren't going to accept this as-is.

Once a developer becomes competent to submit changes without needing that level of critique up-front, you move them to a core committer role, and now you're back to the "no review" case above, because the rest of the committers on that project have better things to do than constantly second-guess their peers' work.

Well where I work we use kind of distributed code review process where the entire development team reviews all of the PRs (generally one or two members per PR) which, in my experience, hasn't been pathological and seems to make good use of GitHub's code review feature, so I don't think it has to be only good for fending off bad PRs, at least it's not that way at our company.

It seems like this was hack only partly a failure of the code review process but I do believe immutability would help by showing which code was known to be directly contributed by Jia Tan (which squashing obfuscates). Also, it might be easier to find other back doors by inspecting their individual commits (at least you could focus on those first and then the others).

Unfortunately GitHub's review process is for source code and not releases and part of the attack code was in a release tarball and not in the source tree:

https://www.akamai.com/blog/security-research/critical-linux-backdoor-xz-utils-discovered-what-to-know

From what I've read, the attacking code in the release tarball used weaknesses contributed into the code, like maybe a failure to guard against buffer overflow (I'm not sure which kind of attacks were actually used). These weaknesses were where code review could have helped but as noted, the project was already floundering, despite being a very useful and pervasive tool.

Automatically built releases could help guard against this particular problem because at least the building code would be in source control (in the GitHub workflow code).

As an aside: popular OSS projects that are floundering look like huge security risks anyway. Organizations with deep pockets would do well to contribute to these so they can reduce risk.

the entire development team reviews all of the PRs

That's approximately how it works here in the Fossil project, only on an ad hoc basis, with only those interested in a given change looking it over. The big risk here is a change that exactly one competent coder wants, with no one else able to review the change caring enough about it to take the time to study it critically.

It's the inverse of the bikeshedding problem. If the change is trivial to the point that everyone can have an opinion, you'll get a huge discussion. If it's complicated and niche, it'll pass without comment.

Guess which class binary blobs of bogus "test data" for xz fall into?

Organizations with deep pockets would do well to contribute to these so they can reduce risk.

In principle, that's what the Fedora → CentOS Stream → RHEL chain is supposed to accomplish, with each step along the chain reviewing what upstream has done to decide whether to push the new release.

What actually happened is that the xz project pushed this bogus new release and it landed in Fedora Rawhide in short order. Had the resulting RPM remained covert, is there any reason to suppose it would have been caught before the Fedora Release process?

Given the de-funding of CentOS, would it have been caught at that level if not?

And then we get to the point where the money comes in, way at the end of this chain, in RHEL. Do their release engineers double-check everything that came in, or do they simply go, "Well, it got pushed into the Fedora $OLDVERSION we set a feature-freeze on 3 years ago, and no one's complained, so it must be good?"

If that were the way of it, why was CentOS forked almost immediately after Red Hat tried pushing everyone to RHEL? Easy answer: people wanted to continue not paying Red Hat $$$ per server for support. It's more cost-effective to send a fraction of that money to a rebuild project like AlmaLinux or Rocky than to pay the first party project what they'd need to provide active code reviewers for all the packages they maintain.

Essentially the same thing happens on the Debian/Ubuntu side; people deploy Debian by the zillions of instances to their clouds in order to avoid Canonical's evil grasping attempts to get everyone onto a Pro subscription.

If I knew more about how SuSE worked, I'm sure I'd identify the same dynamics there, too.

Everyone wants a free ride.

BTW I've found this analisys by Dirk Müeller from OpenSUSE to be of iterest: besides the backdoor itself, he analyzes how downstream distros have handled the bogus updates.