Proposal

Fossil should have a feature similar to Git submodules except with all of the sharp corners knocked off.

Usage

Let's start with a concrete example of how this feature should work in practice:

  $ mkdir -p ~/src/mainproj/trunk
  $ cd ~/src/mainproj/trunk
  $ fossil open ~/museum/mainproj.fossil        # trunk == v2 in development
  $ mkdir lib
  $ cd lib
  $ fossil open --subrepo ~/museum/lib.fossil 
  $ mkdir ../../v1                              # Also check out v1 stable branch.
  $ cd ../../v1
  $ fossil open ~/museum/mainproj.fossil v1     # Doesn't open lib because that...
  $ mkdir lib                                   # ...subrepo is not yet associated...
  $ cd lib                                      # ...with the v1 branch.
  $ fossil open --subrepo ~/museum/lib.fossil 2017-04-01    # old stable version

Contrast with Nested Opens

We already have fossil open --nested, so why do we need fossil open --subrepo?

The primary difference is that --nested doesn't create any tie between the nested repo and the repo in its parent directory. The --subrepo option associates the branch checked out in the parent directory with any version string Fossil understands that is valid within the subrepo, using the same rules open currently uses for its ?VERSION? parameter. This allows the parent repo to either:

1. track trunk in the subrepo
2. track some named branch in the subrepo
3. be tied to some tag in the subrepo
4. be tied to some particular checkin ID in the subrepo
5. be tied to the checkin in the subrepo nearest to a given timestamp

The third option may be particularly interesting to some Fossil users: a repo holding helper libraries might have a stable tag that gets updated each time the libraries are considered stable. Updating a parent repo after the tag moves will update to the latest stable version of the libraries as well.

The --subrepo option has several advantages over --nested:

• fossil up will update subrepos as well.
• fossil ci will check for changes in subrepos in depth-first order; see below.
• fossil open will automatically open subrepos as well, cloning them first if necessary; see below.

Checkin Workflow

If the contents of lib/* are modified in the above scenario, the behavior of fossil ci at the parent level depends on which checkout directory you did it in:

• If you change ~/src/mainproj/trunk/lib/*, a checkin will become the tip of trunk in the lib repo, since we didn't specify a version when opening it as a subrepo.

• If you change ~/src/mainproj/v1/lib/* instead, you'll create a fork in the lib subrepo rooted at the oldest checkin made to it on April Fool's Day 2017.

If you cd into the subrepo's checkout directory, Fossil commands given there will affect only the subrepo, just as with --nested. This allows you to work on the subproject alone for a while, to temporarily switch it to another VERSION, bisect it, etc.

If you create a branch from a one that already has one or more associated subrepos, Fossil clones the configuration to the new branch:

  # fossil ci --branch feature-branch        # new branch gets lib subrepo association

Parent Repo Open Workflow

Having associated one or more branches in the parent repo with one or more subrepos, opening the parent repo will also create the subdirectory for each subrepo and open the specified VERSION there.

If there is already a local clone of the subrepo on the local machine, based on the URL stored in the parent repo's Fossil DB, it opens from that clone, else it makes a clone somewhere sensible. I'd suggest making the clone alongside the parent project's clone, named after the subrepo.

Update and Sync Workflow

In normal operation, update and sync is done from the parent repo level, and is transparent to the user, except for the extra time it takes.

To change a subrepo to another version, with the association with the parent repo's current checked-out branch and the subrepo's version being updated:

  $ fossil up --subrepo 2017-03-15 lib      # roll back to earlier stable version

Without the --subrepo option, it just updates the subrepo directory without changing the association: saying fossil up at the parent level will roll it forward to either trunk or 2017-04-01, depending on which parent checkout directory we're in, under the usage scenario at the top of this proposal.

Updating at the parent level may change the subrepo contents:

  $ cd ~/src/mainproj/trunk
  $ fossil up feature-branch                # changes lib accordingly

To change the clone URL for a subrepo:

  $ cd subrepo ; fossil sync https://example.com/subrepo

Rollback Behavior

The initial version of this feature does not have to attempt to handle inter-repo rollbacks. It can be implemented with subprocesses, much as fossil all currently does.

If you have changes in both the parent repo and in one of it subrepos, then give a checkin command at the parent level:

• A failure down in a submodule stops processing. The changes to neither project are checked in: the subrepo because it was rolled back by the failure, and the parent because the child fossil instance died with a nonzero exit status due to the failure at the subrepo level. (If it were otherwise, the parent repo could have a checkin that depends on a change at the subrepo level that did not get checked in.

• A failure at the parent level results in the checkin to the leaf being kept, with the parent repo changes not yet checked in.

Inter-repo rollbacks might be neat, but it's an edge case that doesn't need to be solved any time soon.

DB Requirements

Fossil needs to remember only a few things per branch to support this:

1. The list of subrepos.
2. The VERSION string specified for each subrepo, if any.
3. The relative path of each subrepo within the parent.
4. The clone URL for each subrepo, with which it can look up the local clone file using the underpinnings of fossil all .

I've thought about whether this should be stored in the blockchain, and I think the answer is, "No."

The clone URLs definitely should not be stored in the blockchain: repo URLs may change, if only because a site stops allowing HTTP clones, so if the URL were in the blockchain, moving the repo would prevent you from checking out old versions of the parent project.

That then calls into question whether any of the rest should be in the blockchain, since now we've implicitly got a new DB table mapping branch names to subrepo clone URLs. Why not store the rest of it there, too? The only reason I can think of is that you might want subrepos to be tied to individual checkins, in the same way that branch names are currently handled.

That seems like a bogus argument to me, though: I'm happy associating a subrepo with a branch. I see no reason to make it work with individual checkins. The set of subrepos should not be changing that rapidly. If you need a particular checkin to have a different set of subrepos than its ancestor checkin, create a new branch for that checkin.