Can't import two git repos with `--attribute` option to `fossil import`

(1) By James Cook (falsifian) on 2023-06-18 02:09:46 [link] [source]

I'm thinking of consolidating multiple git repos into one fossil repo, and ran into the following issue.

It seems if I run fossil import twice, and pass the --attribute option both times, I see the following error the second time:

SQLITE_ERROR(1): table fx_git already exists in "CREATE TABLE fx_git(user TEXT, email TEXT UNIQUE);"
Database error: table fx_git already exists: {CREATE TABLE fx_git(user TEXT, email TEXT UNIQUE);}

By the way, if fossil import is supposed to store the arguments to --attribute for future reference (I'm just guessing based on what I see here), that should really be documented in my humble opinion. I am currently imagining a nightmare scenario where I just can't figure out why fossil import is renaming a@example.com to user foo, and it not even occurring to me that fossil might remembering it from a previous invocation...

Here is a script to reproduce:

#!/bin/sh
set -e

mkdir example;cd example

fossil init test.fossil

echo first import...
fossil import --git -i --attribute 'falsifian@falsifian.org falsifian' test.fossil <<EOF
blob
mark :1
data 13
Hello world!

reset refs/heads/main_a
commit refs/heads/main_a
mark :2
author James Cook <falsifian@falsifian.org> 1686928435 +0000
committer James Cook <falsifian@falsifian.org> 1686928435 +0000
data 15
Create a file.
M 100644 :1 a_file

commit refs/heads/main_a
mark :3
author James Cook <falsifian@falsifian.org> 1686928445 +0000
committer James Cook <falsifian@falsifian.org> 1686928445 +0000
data 17
Rename the file.
from :2
R a_file a_renamed_file

EOF

echo second import...
fossil import --git -i --attribute 'falsifian@falsifian.org falsifian' test.fossil <<EOF
blob
mark :1
data 13
Hello world!

reset refs/heads/main_b
commit refs/heads/main_b
mark :2
author James Cook <falsifian@falsifian.org> 1686928435 +0000
committer James Cook <falsifian@falsifian.org> 1686928435 +0000
data 15
Create a file.
M 100644 :1 a_file

commit refs/heads/main_b
mark :3
author James Cook <falsifian@falsifian.org> 1686928445 +0000
committer James Cook <falsifian@falsifian.org> 1686928445 +0000
data 17
Rename the file.
from :2
R a_file a_renamed_file

EOF

echo done  # never reached

(2.1) By mark on 2023-06-22 13:56:43 edited from 2.0 in reply to 1 [source]

ETA: This fix has landed on trunk, and the import docs have also been updated to mention attributions are persistent. Although documented here https://fossil-scm.org/home/doc/trunk/www/inout.wiki, it should be in the import help text too.

Thanks for the repro recipe, James!

Thanks for reporting this, James. Can you please let me know if the below diff fixes this for you?

wrt the attribute data being persisted, that is documented here: https://fossil-scm.org/home/doc/trunk/www/inout.wiki

Alternatively, the --attribute option can be passed to have all commits by a given committer attributed to a desired username. This will create and populate the new fx_git table in the repository database to maintain a record of correspondent usernames and email addresses that can be used in subsequent exports or incremental imports.

However, I agree that this should be mentioned in the import help text too.

Index: src/import.c
=======================================================================
hash - 26bf6db4a5a11bbec1ddde3be392cade9aecdf3a8fe7d2ef39cc54f755debe7b
hash + 90b15eb17abf701f27c86988415c9d3de0db54e28cc813671b07ff451662ce50
--- src/import.c
+++ src/import.c
@@ -1957,9 +1957,11 @@ void import_cmd(void){
     if(ggit.nGitAttr > 0) {
       int idx;
       db_unprotect(PROTECT_ALL);
-      db_multi_exec(
-        "CREATE TABLE fx_git(user TEXT, email TEXT UNIQUE);"
-      );
+      if( !db_table_exists("repository", "fx_git") ){
+        db_multi_exec(
+          "CREATE TABLE fx_git(user TEXT, email TEXT UNIQUE);"
+        );
+      }
       for(idx = 0; idx < ggit.nGitAttr; ++idx ){
         db_multi_exec(
             "INSERT OR IGNORE INTO fx_git(user, email) VALUES(%Q, %Q)",

(3) By John Rouillard (rouilj) on 2023-06-18 04:39:15 in reply to 2.0 [link] [source]

Is https://fossil-scm.org/home/doc/trunk/www/inout.wiki up to date? It shows the use of fossil export --git ...| fast-import which I thought was replaced with fossil git export [MIRROR] (no need for fast-import) a bit ago.

Apparently, there is also a fossil git import MIRROR but it is not documented according to fossil help git.

(4) By mark on 2023-06-18 05:33:31 in reply to 3 [link] [source]

fossil git import is/was intended to replace fossil import --git like fossil git export replaced fossil export --git; hence, the TBD in the fossil help git import subcommand. Development has since stalled (or been abandoned) though. But, yes, the fossil export --git reference in that doc should be replaced with the new implementation.

(5.1) By James Cook (falsifian) on 2023-06-23 02:12:29 edited from 5.0 in reply to 2.1 [link] [source]

Thanks mark, that fixes it for me!

Personally I am not a big fan of Fossil persisting the mappings, but it looks like I can just drop the fx_git table to get the behaviour I want, so it doesn't really matter.

Maybe it's just that I haven't used Fossil for long enough to get used to the way it likes to persist a bunch of stuff in my local repository clones. So, please don't take the following arguments for not persisting the state to mean I think I know better than the Fossil developers; I'm putting them here in case I'm not the only one who's bugged by it, in which case maybe they can be a conversation starter.

I find that the less state a tool has, the easier a time I have understanding what it does and incorporating it into scripts.
I like version-controlling things, so I'd like to keep the mappings in a text file for that reason.
Unless I'm mistaken there's no way to edit or delete the mappings already inserted, other than running fossil sql commands.

(Settings being in the database rather than a version-controllable text file also bugs me a bit. But I'm certainly not asking Fossil to switch directions on this on my account, since it seems to be working well for people!)

(6) By Stephan Beal (stephan) on 2023-06-23 05:56:50 in reply to 5.1 [link] [source]

Maybe it's just that I haven't used Fossil for long enough to get used to the way it likes to persist a bunch of stuff in my local repository clones.

Sidebar which you might already know: fx_... tables never sync between repos. The fx_ prefix was specifically added for "fossil extensions" and can contain arbitrary stuff unrelated to the SCM. During a rebuild, fossil will drop any tables which aren't its own unless they start with that prefix.

(Settings being in the database rather than a version-controllable text file also bugs me a bit. But I'm certainly not asking Fossil to switch directions on this on my account, since it seems to be working well for people!)

It has both, but not all settings are (or should be) versionable. Some settings are most definitely user preferences, not project preferences, or are security-relevant, so are not versioned.

(7) By James Cook (falsifian) on 2023-06-23 14:20:13 in reply to 6 [link] [source]

Sidebar which you might already know: fx_... tables never sync between repos. The fx_ prefix > was specifically added for "fossil extensions" and can contain arbitrary stuff unrelated to the > SCM. During a rebuild, fossil will drop any tables which aren't its own unless they start with that prefix.

Thanks. I figured it wouldn't be synced, but I didn't know about the fx_ prefix being protected.

It has both, but not all settings are (or should be) versionable. Some settings are most definitely user preferences, not project preferences, or are security-relevant, so are not versioned.

Sorry, I wasn't clear about this. I mean that text files are easier to manage, so I like it when configuration is via plain-text files. Things like my text editor's settings, shell aliases, etc. I keep those in a git repo and have scripts to keep them in sync between the repo and different computers I use. I think I would like to keep my private fossil settings similarly in sync / tracked, though experience could change my mind.

Now that I've said that, I realize there's nothing stopping me from dumping my local fossil settings into text files and extending my script to sync the settings between the text files and the actual fossil repos. So I guess I don't have much of a complaint here.