Fossil Forum

filesystem versus database, least interaction
Login

filesystem versus database, least interaction

filesystem versus database, least interaction

(1) By anonymous on 2023-01-08 10:43:11 [link] [source]

Lower level functionality, (without any pre-filtering with command line file
systems tools).

Since sqlite (fossil databases are such) abstracts the filesystem, can a fossil repository, alas sqlite database be created based on a list of files,
regardless their location on the file system, as such, do away with the notion of a checkout directory, which is filesystem related terminology and mental construct. That would be an elegant addition to the binary?

Many thanks,

(2) By Stephan Beal (stephan) on 2023-01-08 11:35:08 in reply to 1 [link] [source]

Since sqlite (fossil databases are such) abstracts the filesystem, can a fossil repository, alas sqlite database be created based on a list of files, regardless their location on the file system, as such, do away with the notion of a checkout directory,

The short answer is no. The restriction of "has a checkout directory" is very much baked in to fossil and is fundamental to many of its algorithms.

Allowing fossil to operate outside of a given checkout is inherently a major, and very real (not academic or hypothetical), security risk. If you were to open such a repository from some complete stranger, it might overwrite your /etc/hosts, your ~/.bashrc, or any other file.

That would be an elegant addition to the binary?

Perhaps to a binary, but not to fossil's binary (IMO). SCMs are designed to house individual projects, by and large source code, and not arbitrary files from arbitrary places in a single repository which spans a whole root filesystem. We cannot escape the "mental construct" of a filesystem so long as fossil lives within one and serves from housed in one.

(5) By Warren Young (wyoung) on 2023-01-08 12:29:36 in reply to 2 [source]

a checkout directory…is fundamental to many of its algorithms.

I believe the OP is thinking on the /fileedit level. From that point of view, what can't you do that you want to?

I'll predict one and shoot it down: edit multiple files, then check them all in at once, as a single commit. That merely sounds like a wish for a web-based staging area or bundle mechanism, which could be done using the same local storage methods you have in /fileedit already. Once you have all your edits staged, you commit them all at once.

I'm not trying to arm-twist you into doing the work, just trying to confirm my sense that it's merely contingent on Fossil's history, not truly fundamental. If I'm right, it could be done if one were sufficiently motivated, using methods as in my post #3.

a major, and very real (not academic or hypothetical), security risk

No more than is posed by /fileedit, which as far as I'm aware, is as strong security-wise as Fossil generally: its RBAC, its SQL verifier, the new DB read-only locking feature… That in turn is all protected by your local TLS setup, plus things like chroot, containers, etc.

If you were to open such a repository from some complete stranger, it might overwrite your /etc/hosts, your ~/.bashrc, or any other file.

GNU tar solved that in 1989. Short of unpacking the archive at the filesystem root — which the OP doesn't want to do anyway with his wish to avoid checkouts — you have to go out of your way to force it to not do this, by giving it the --absolute-paths flag.

I can't be bothered to find out when AT&T and BSD tar got this ability, but it's table stakes today.

I spent some time skimming the Fossil code to find out whether it does indeed protect itself against such things, and as far as I can tell, you have to open the repo at the filesystem root to run into this problem since it canonicalizes all paths under the checkout root.

(7) By Stephan Beal (stephan) on 2023-01-08 13:18:03 in reply to 5 [link] [source]

No more than is posed by /fileedit, which as far as I'm aware, is as strong security-wise as Fossil generally: its RBAC, its SQL verifier, the new DB read-only locking feature… That in turn is all protected by your local TLS setup, plus things like chroot, containers, etc.

The risk inherently posed by /fileedit is limited to a single checkout, however. Do away with a checkout, such that a single repo can manage anything across a system, and the risk explodes into arbitrary directories. The sandbox of a checkout is as much a security abstraction as it is an organizational one.

(Pardon my brevity and likely lack of future responses - finalizing packing up for a move in the morning.)

(9) By Warren Young (wyoung) on 2023-01-08 13:26:32 in reply to 7 [link] [source]

If the OP wants Fossil as the root filesystem, I'd say the lack of file permissions is a bigger problem than any of that.

(6) By anonymous on 2023-01-08 12:44:39 in reply to 2 [link] [source]

You have looked at the problem from all angles. I fully agree. 

The defective element above the kernel is the filesystem. The filesystem should be able to take care of versioning, backup, "snapshots". The relationship between the hardware and the kernel should be "open source". That would make head-way and eliminate the higher constructs and "frame-works" ad infinitum. A relationship of Gigabytes of software to a few Megabytes of content (more often Kilobytes) is preposterous.

The same goes for "mark-up" on the level of the single file, the relationship between man and machine, if it is to be one of plain text, without avoidable layers of scripting, high-level "programming", markup/downtown that can be ignored by the machine is often a step foreward. Pdf use and html&scripting is such a layer of complexity that caters to the human, and complicates to the machine side.

The concept of a database suffers from above, i guess you developers at sqlite are aware of it, hence your argument of a "single file" repository, which does away with a lot of mental hubris. Note, open up the database, ...and butt upon a file structure.

(8.1) By Warren Young (wyoung) on 2023-01-08 13:28:49 edited from 8.0 in reply to 6 [link] [source]

The filesystem should be able to take care of versioning, backup, "snapshots".

Many do, and some have for decades. The premier modern example is ZFS, coupled with its send/recv features, now cloned by btrfs.

There's also APFS + Time Machine in the Apple world, ReFS + VSS in the Microsoft world, etc.

The main thing you lack in these relative to Fossil is a commit commenting mechanism. Some of them allow a hierarchy of versions, etc.

eliminate the higher constructs and "frame-works" ad infinitum

What is Fossil in this context but another framework, an abstraction of filesystem contexts?

SQLite is on the same order of complexity as a filesystem, which you want to replace.

What've you bought?

A relationship of Gigabytes of software to a few Megabytes of content (more often Kilobytes) is preposterous.

The tradeoff is faster development time, without which you wouldn't have as much useful software as you do, including Fossil, which depends on:

  • A whole pile of development tools (compilers, editors, build tools, a Tcl interpreter…)
  • The browser's JS interpreter and HTML layout engine at runtime
  • And at both development and runtime, much of the OS comprising those gigabytes

If you look at Fossil as a 4-6 MiB executable while ignoring all that, I wonder how much you think Fossil should adopt into its core. The plan is beginning to look distinctly inelegant.

markup that can be ignored by the machine is often a step foreward

I'll agree with you this far. I've stopped using word processors, for the most part. When I need to produce PDFs or paper documents beyond the capability of Markdown and a sensible rendering engine + style sheet, I jump to page layout software.

html&scripting is such a layer of complexity

Can we infer from this that you don't find /fileedit or the linked VSCode Online options appealing?

If not them, then what do you propose? A Vim plugin that edits files from the repo instead?

open up the database, ...and butt upon a file structure.

I'm not parsing your English well, but if I understand it, I want you to contemplate this: what is a filesystem if not a hierarchical database indexed by paths?

(10) By anonymous on 2023-01-08 15:28:52 in reply to 8.1 [link] [source]

Was aware of zfs, btrfs, and "snapshots". Being on OpenBSD, never tried it out.

Commit messages, adding one's own meta-notes is a definite advantage. To the expense of another layer.
File,  the abstraction of the inode, on the file-system, should have a header that allows for writing to it, and reading from it in case
to do one better. Not the case to my knowledge in any os/filesystem?!

"What is Fossil in this context but another framework, an abstraction of filesystem contexts?

SQLite is on the same order of complexity as a filesystem, which you want to replace."


Indeed, the theoretical NO, creating another layer of software.


"A Vim plugin that edits files from the repo instead?"


In the case of Fossil, as is, there is the checkout and the main functionality outside the web interface.

To me the web interface caters to the third parties, as a forum mainly.

It all seems obvious, but it is not, computing as designed today, within technological limits, and the distance

between open (non-existing) hardware instructions and kernel, then filesystem tools  up to  Fossil is  not the  best design

for progress. I think historically every-one was proprietary driven, to secure a scrap of the loot. The evolution of software

goes into the direction of hubris.


Thanks for your insights, i do appreciate this poking into the inners,

(11) By Warren Young (wyoung) on 2023-01-08 16:45:18 in reply to 10 [link] [source]

Not the case to my knowledge in any os/filesystem?!

HFS had resource forks for metadata like this in 1985, since added to many other filesystems, and Linux extended attributes allow 4 to 64 kiB of data per attribute, depending on the actual FS in use. That's plenty for a checkin comment.

In the case of Fossil, as is, there is the checkout

You started this thread wanting to get away from checkouts. So, how do you expect to edit a file, exactly?

(12) By anonymous on 2023-01-08 18:12:28 in reply to 11 [link] [source]

Thanks for your pointers, will look into these when a linux system is at hand. OpenBSD has (no longer) support for writing to file headers.

"...expect to edit a file"

Made myself misunderstood. Editing files, of course, outside the web interface, understanding where addremove adds them in the database was more important, so the editor within a checkout can be used (vim or anything else) knowing where they would appear in the web  interface.

My question should have been, howto have cli fossil inventory specific files that are of interest to version control (certain files in /etc, not all, a ~/.bashrc, a  ~/.myscripts directory, the $DOCUMENTROOT of apache. These are just examples, but in all cases these files are within different directories that only come together at the root of the file system, so to put them under version control by putting the "/" root of the filesystem as the checkout would be unpractical (a lot of noise).

Thus the idea is to abstract the checkout directory to something as: "fossil addremove" "the files/directories as listed in the configuration file particular to that repository regardless of their different locations on the tree".

A xapian search repository/database allows configuring for this as an example of similar filesystem (parts), up to atomic single files, then can be added to a database.

(13) By Warren Young (wyoung) on 2023-01-08 18:35:06 in reply to 12 [link] [source]

certain files in /etc

Simple answer, then: don't. Fossil doesn't track file permissions.

You can find this asked and answered multiple times in the archives here.

There are other tools that do this properly.

(3) By Warren Young (wyoung) on 2023-01-08 11:48:50 in reply to 1 [link] [source]

You can remove all your checkouts and have such a database.

The trick is, what can you do with it now that you have it, without creating a new checkout?

Most of the Fossil functionality can be done through the web UI, one way or another, including file editing thanks to Stephan's /fileedit feature. It's useful, but IMO it's more an emergency or contingent-use feature, not something you want to use every day. Even 1979 vi is a more capable text editor than what you get with browser <textarea> tags.

The bigger DVCSes have features like github.dev to strap a better editor onto their web UI, but the Fossil project doesn't have GitHub's staff to do such integrations. We have volunteers, but are you the one who's going to do this for Fossil? If not you, then who, and why?

Inversely, there are independent web-based editors like vscode.dev that know how to reach out and edit files on a remote repo without cloning it, but I don't know how you'd get them to add Fossil support. There's a VSCode Fossil plugin, but I doubt that helps here. It probably requires someone from Microsoft to do the low-level integration. How do you get them to do that when they're all-in on Git these days?

That aside, these mechanisms aren't "elegant additions to the binary." They're external integrations that make use of the back-end DVCS's existing capabilities. Fossil should be able to do the same, at least in principle, but the hard work needed to do it isn't in the Fossil binary.

(4) By Stephan Beal (stephan) on 2023-01-08 12:00:02 in reply to 3 [link] [source]

The bigger DVCSes have features like github.dev to strap a better editor onto their web UI, but the Fossil project doesn't have GitHub's staff to do such integrations. We have volunteers, but are you the one who's going to do this for Fossil?

Little-known trivia, though: there is a hook in place to replace the admittedly horrid /fileedit and /wikiedit editor widget with one's own. It would have to be loaded via the site skin, and it's never been done "outside of the lab," but we do have a hook in place for it:

src:/doc/trunk/www/fileedit-page.md#editor

Even so, repository still has a virtual root directory which becomes the root of any checkout, so a user cannot completely eliminate the idea of "some top-level directory" in fossil. Whether that top is a repository root or checkout root is arguably irrelevant, as a checkout is effectively just a different representation of a repository's root.

(14) By KIT.james (kjames3411) on 2023-01-15 05:47:44 in reply to 1 [link] [source]

If I were you I would play with fuse(4)