Push/Write capability ineffective having switched transports
(1.2) By sebyte on 2025-04-17 20:58:32 edited from 1.1 [link] [source]
Having switched transports (from HTTPS to SSH), I am no longer able to push.
$ fsl push
Push to ssh://XXX/fossil/museum/fossil-sebyte.fossil
Round-trips: 1 Artifacts sent: 0 received: 0
Error: not authorized to write
Round-trips: 1 Artifacts sent: 0 received: 0
Push done, wire bytes sent: 705 received: 212 remote: XXX
Hmm... have my capabilities changed on the remote?
$ whoami
sebyte
$ remote=$(fsl remote | cut -d/ -f4-)
$ sql="SELECT cap FROM user WHERE login='sebyte';"
$ ssh $FOSSIL_HOST sqlite3 $remote \"$sql\"
sy
Looks good to me. I should be able to write. (I was before I switched transports).
How many commits remain unpushed?
$ fsl sql
SQLite version 3.50.0 2025-04-10 10:18:07
Enter ".help" for usage hints.
sqlite> select count(*) from unsent;
0
Eh? That figure should be at least 1. Let’s see what we know about the parent of the current commit.
$ fsl info | grep ^parent
parent: 9a2d2a05ceb9877cb668dddb0d52e0a7e23a61a7 2025-02-25 13:55:54 UTC
$ fsl info 9a2d2a05
hash: 9a2d2a05ceb9877cb668dddb0d52e0a7e23a61a7 2025-02-25 13:55:54 UTC
parent: 7c861cee5b2cbed7b0df6d9ba18bf6be37dc83cb 2024-06-04 16:21:08 UTC
merged-from: 6782c579af31ad359a77e896cd1f10c09f9ae526 2025-02-24 07:44:19 UTC
child: 726a9cdc5acc091283b26502a72584bc31cf4a86 2025-04-15 16:14:53 UTC
tags: sebyte
comment: Merge trunk. (user: sebyte)
Now let’s run the same command on the remote repository. (Command rfsl
is a simple bash function which runs a Fossil command on the remote repo, using ssh
).
$ rfsl info 9a2d2a05
hash: 9a2d2a05ceb9877cb668dddb0d52e0a7e23a61a7 2025-02-25 13:55:54 UTC
parent: 7c861cee5b2cbed7b0df6d9ba18bf6be37dc83cb 2024-06-04 16:21:08 UTC
merged-from: 6782c579af31ad359a77e896cd1f10c09f9ae526 2025-02-24 07:44:19 UTC
tags: sebyte
comment: Merge trunk. (user: sebyte)
No child! Checkin 726a9cdc
(the checkin I want to push) does not exit on the remote yet the unsent
table is empty?
In summary, I think I’ve unearthed two bugs here:
- By switching transports from HTTPS to SSH (same username for both), I've lost the ability to push/write. (Pulling works fine).
- The unsent table is not being populated in the manner it should.
UPDATE: Just noticed that page /setup_ucap_list
indicates that 2 users have capability i
while page /setup_ulist?with=i
lists only one user; sebyte
. This corroborates my sense that there are in fact two sebyte
s; one HTTPS sebyte
and one SSH sebyte
. (SSH sebyte
lacks cap i
and is not listed).
For the skeptical/curious, here’s the definiton of bash function rfsl
:
$ type rfsl
rfsl is a function
rfsl ()
{
local remote_raw=$(fossil remote);
if [[ $remote_raw =~ ^ssh ]]; then
local remote_path=$(echo $remote_raw | cut -d/ -f4-);
ssh $FOSSIL_HOST fossil "$@" -R $remote_path;
else
echo Not a Fossil checkout or the transport is not SSH. Abort.;
fi
}
(2) By Richard Hipp (drh) on 2025-04-17 20:32:55 in reply to 1.0 [link] [source]
I'm sorry you are having trouble. It works for me. I use this feature every day, multiple times per day. I've never had any issue with it. I do not have an ideas about how it is going wrong for you.
(3) By sebyte on 2025-04-17 20:54:21 in reply to 2 [link] [source]
Oh dear. Does the edit/update to my original post about there (effectively) being two sebyte
s offer any clues?
(5) By Andy Bradford (andybradford) on 2025-04-17 21:03:55 in reply to 3 [link] [source]
> Does the edit/update to my original post about there (effectively) > being two sebytes offer any clues? Unless you're doing something fancy with SSH and Fossil, then it doesn't matter what the Fossil permissions are or what Fossil users exist in the remote repository. By default, Fossil over SSH uses only file permissions, not Fossil user permissions, so your "SELECT cap FROM user WHERE login='sebyte';" is irrelevant. How have you configured SSH access to the Fossil repositories? Andy
(4.1) By Andy Bradford (andybradford) on 2025-04-17 23:24:21 edited from 4.0 in reply to 1.0 [link] [source]
> Having switched transports (from HTTPS to SSH), I am no longer able to > push. In their default uses with Fossil, SSH and HTTP(S) have entirely different permissions models. With HTTP(S), Fossil is able to enforce it's own RBAC. The default operation with SSH and Fossil, on the other hand, relies entirely upon the permissions on the file. Switching transport may or may not be a simple thing depending on the environment and how it's configured. Looking at this output: Push to ssh://XXX/fossil/museum/fossil-sebyte.fossil So that would reference a "relative" path that is in the home directory of the user that you're using with SSH. For example, if the username is sebyte, then that path would amount to: ~sebyte/fossil/museum/fossil-sebyte.fossil Is this the path that you expect? Does the SSH user that you're using have permissions to write to that file and the directory in which it is located? The most likely cause is a permission problem. If it's not, then to get a little more information you could run: fossil push --sshtrace --httptrace The sshtrace output will be in your terminal. The httptrace output will be in a set of files in the directory where you run the command. If that's still insufficient, you can get extra verbose SSH output with: fossil push --once --sshtrace --httptrace --ssh-command 'ssh -e none -vvv' The -vvv will produce a lot of output (most of which may be unhelpful) so be prepared. Andy
(6.1) By sebyte on 2025-04-18 11:27:40 edited from 6.0 in reply to 4.1 [link] [source]
By default, Fossil over SSH uses only file permissions, not Fossil user [capabilities], so your "SELECT cap FROM user WHERE login='sebyte';" is irrelevant.
Understood.
~sebyte/fossil/museum/fossil-sebyte.fossil
Is this the path that you expect?
Yup.
Does [sebyte] have permissions to write to that file and the directory in which it is located?
Yup. The directory has mode 0755
, the file has mode 0644
and the owner is sebyte
in both cases. (This is the first thing I tested).
The most likely cause is a [file] permission problem.
I agree, but I can do this, for example:
$ ssh $FOSSIL_HOST ls -l fossil/museum/fossil-sebyte.fossil
-rw-r--r-- 1 sebyte sebyte 360013824 Apr 18 08:00 fossil/museum/fossil-sebyte.fossil
$ ssh $FOSSIL_HOST echo foo \> fossil/museum/bar.txt
$ ssh $FOSSIL_HOST cat fossil/museum/bar.txt
foo
Here are the traces:
$ fsl push --sshtrace --httptrace
Push to ssh://XXX/fossil/museum/fossil-sebyte.fossil
RUN ssh -e none -T -- XXX fossil test-http fossil/museum/fossil-sebyte.fossil
Got line: [Status: 200 OK]
Got line: [Cache-control: no-cache]
Got line: [X-Frame-Options: SAMEORIGIN]
Got line: [Content-Type: application/x-fossil-debug]
Got line: [Content-Length: 74]
Got line: []
Reading 74 bytes with 0 on hand... Got 74 bytes
Round-trips: 1 Artifacts sent: 0 received: 0
Error: not authorized to write
Round-trips: 1 Artifacts sent: 0 received: 0
Push done, wire bytes sent: 1005 received: 208 remote: XXX
/***** Subprocess 3894 exit(0) *****/
$ cat http-request-1.txt
POST fossil/museum/fossil-sebyte.fossil HTTP/1.0
Host: XXX
User-Agent: Fossil/2.26 (2025-04-15 16:14:53 [726a9cdc5a])
X-Fossil-Transport: SSH
Content-Type: application/x-fossil-debug
Content-Length: 792
pragma client-version 22600 20250415 161453
push 53a4a3f8301634bfd961387988da71da0aee00c9 CE59BB9F186226D80E49D1FA2DB29F935CCA0333
igot 0b01337c706b762ce81e3d5849cc4413cd041bd2262f3087c555ba89872d60f4
igot 2b96941c4c924e56246d673a30d5ef5146824dfd7361af503c55bbf66fc4eac8
igot 2d3ace5a9fb4de5cd37ce2fdff0012cb438986ddeb3f866d172b7a3a3b453056
igot 52b3992f0aeaaf17b8a4ef7cb4e1417baa1d8ae57f01b681887a8452a19e3a4d
igot 726a9cdc5acc091283b26502a72584bc31cf4a86c7313526c72fbc3be8a9b842 <--- Checkin that needs pushing.
igot 97cc76f99d67451246f5f5bcc6b7bcd6b9f5e3b7bdf03fbfe68c4f695b90ba67
igot ba0242f8d1748b8ba585dcab598e9e52088a4a751dc1b75b334670c2b88ef936
igot bf1e123edfba0a212a49f7bc0858d752a677ee3cdf5ce107dd25f42e53c08b26
pragma ci-unlock 5359737ff569218b1903247629ec6e5c3e0d95f8
# A2947ADA44AF7288B29B272B668C93090DFFE281
$ cat http-reply-1.txt
Status: 200 OK
Cache-control: no-cache
X-Frame-Options: SAMEORIGIN
Content-Type: application/x-fossil-debug
Content-Length: 74
error not\sauthorized\sto\swrite
# timestamp 2025-04-18T09:13:38 errors 1
Here is the verbose SSH output (at debug level 2) with obviously uninteresting lines elided.
$ fsl push --sshtrace --httptrace --ssh-command 'ssh -e none -vv'
Push to ssh://XXX/fossil/museum/fossil-sebyte.fossil
RUN ssh -e none -vv -T -- XXX fossil test-http fossil/museum/fossil-sebyte.fossil
OpenSSH_8.4p1 Debian-5+deb11u4, OpenSSL 1.1.1w 11 Sep 2023
[…]
debug1: Next authentication method: publickey
debug1: Offering public key: /etvc/ssh/user-key/hamsi_sebyte_id_rsa.pub RSA SHA256:9XI/p3yHBgSEdKY2tAjVro3JJO1qWC18tgYTW6cx01k agent
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /etvc/ssh/user-key/hamsi_sebyte_id_rsa.pub RSA SHA256:9XI/p3yHBgSEdKY2tAjVro3JJO1qWC18tgYTW6cx01k agent
debug1: Authentication succeeded (publickey).
Authenticated to XXX ([xx.xx.xxx.xxx]:yyyyy).
[…]
debug1: Remote: /etvc/ssh/authorized-keys/sebyte:37: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /etvc/ssh/authorized-keys/sebyte:37: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug2: channel_input_open_confirmation: channel 0: callback start
debug1: Requesting authentication agent forwarding.
[…]
debug1: Sending command: fossil test-http fossil/museum/fossil-sebyte.fossil
[…]
Got line: [Status: 200 OK]
Got line: [Cache-control: no-cache]
Got line: [X-Frame-Options: SAMEORIGIN]
Got line: [Content-Type: application/x-fossil-debug]
Got line: [Content-Length: 74]
Got line: []
Reading 74 bytes with 0 on hand... Got 74 bytes
Round-trips: 1 Artifacts sent: 0 received: 0
Error: not authorized to write
Round-trips: 1 Artifacts sent: 0 received: 0
Push done, wire bytes sent: 1005 received: 208 remote: XXX
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
/***** Subprocess 4339 exit(0) *****/
$ debug2: channel 0: write failed <--- Clue?
debug2: channel 0: chan_shutdown_write (i3 o0 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: send eow
debug2: channel 0: output open -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd eof
debug2: channel 0: rcvd close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 4184, received 3504 bytes, in 0.2 seconds
Bytes per second: sent 21504.8, received 18009.8
debug1: Exit status 0
The line I've highiighted debug2: channel 0: write failed
makes me think SSH itself is experiencing a problem somehow. But then we also have debug1: Exit status 0
which tells me SSH itself is fine.
How have you configured SSH access to the Fossil repositories?
I'm not sure how to answer this question. I use ssh-agent
to store my passphrase locally. SSH gives me access to the Fossil repository in the same way it does every file and folder beneath my home directory (on the remote machine).
(7) By Richard Hipp (drh) on 2025-04-18 13:25:49 in reply to 6.1 [link] [source]
Please try this:
- SSH into the remote.
- CD into a check-out for the repository you are trying to access
- Run:
fossil unset localauth && fossil unset localauth -global
Then retry the SSH push and report back what happens.
(9.1) By sebyte on 2025-04-18 15:37:33 edited from 9.0 in reply to 7 [link] [source]
After unsetting localauth
both locally & globally (on the remote), the push worked!
I can confirm that localauth
was on (locally, value 1
) before unsetting it.
localauth
is definitely the culprit. Thank you! I will investigate further and report back.
(10) By Andy Bradford (andybradford) on 2025-04-18 14:39:34 in reply to 7 [link] [source]
> Then retry the SSH push and report back what happens. It seems that disabling localauth resolved sebytes issue, however, I'm trying to reproduce the problem by intentionally setting localauth on a remote repository and I'm unable to reproduce it: $ ssh remote fossil settings localauth -R /tmp/new.fossil localauth (local) on $ echo $RANDOM >> file $ fossil ci --nosync -m test New_Version: b23e72909e558681ae2c28039e317ee0ab955ae43b0e5ef9a2af62f2e96c5578 $ fossil push Push to ssh://remote//tmp/new.fossil Round-trips: 1 Artifacts sent: 2 received: 0 Push done, wire bytes sent: 754 received: 217 remote: remote $ ssh remote fossil info b23e7290 -R /tmp/new.fossil hash: b23e72909e558681ae2c28039e317ee0ab955ae4 2025-04-18 14:26:52 UTC parent: 1ac996fea6abd456f68dfed6b92f7f0515f3d5db 2025-04-18 14:25:23 UTC tags: trunk comment: test (user: amb) I even tried turning it on globally on the remote host and it still permits me to push. So I'm confused how this could be the issue. The help for localauth says: (1) This setting ("localauth") must be off But clearly it doesn't matter in my case because I'm still able to push, even when it's enabled on the remote repository. Is there some other condition that I'm missing?
(11) By Richard Hipp (drh) on 2025-04-18 14:43:18 in reply to 10 [link] [source]
With the localauth setting on, it uses the permission for whatever the local OS username is. So you will have to disable those permissions too.
(13) By Andy Bradford (andybradford) on 2025-04-18 14:51:18 in reply to 11 [link] [source]
> With the localauth setting on My trouble in reproducing is that "on" is not equivalent to "1". Should it be for this setting? This claims that it's a boolean: https://fossil-scm.org/home/file?ci=trunk&name=src/db.c&ln=4926 But this does not treat it as boolean: https://fossil-scm.org/home/file?ci=trunk&name=src/login.c&ln=1421 Which should be corrected? Thanks, Andy
(12) By Andy Bradford (andybradford) on 2025-04-18 14:44:25 in reply to 10 [link] [source]
> Is there some other condition that I'm missing? Apparently a localauth setting of "1" is not the same as a localauth setting of "on"; when I set it to "1" I can reproduce it. The help says: Setting: "localauth" (default: off) So I asusmed that "on" would be sufficient to enable it, but I guess it really wants "1".
(8.1) By Andy Bradford (andybradford) on 2025-04-18 13:51:36 edited from 8.0 in reply to 6.1 [link] [source]
> debug1: Authentication succeeded (publickey). > Authenticated to XXX ([xx.xx.xxx.xxx]:yyyyy). Ok, so you're using SSH keys. Good to know. What do you have in your remote ~sebyte/.ssh/authorized_keys that permits you to run Fossil on your remote? What I'm looking for is to see if you have any ForceCommands enabled either in /etc/ssh/sshd_config or ~sebyte/.ssh/authorized_keys for the SSH key that you're using for authentication. > The line I've highiighted debug2: channel 0: write failed makes me > think SSH itself is experiencing a problem somehow. Hard to say if this is related to the problem at hand... > debug2: channel 0: read failed > debug2: channel 0: write failed I see these same debug2 messages on my SSH repositories where I have no problem writing: debug2: channel 1: read failed rfd 5 maxlen 32768: Broken pipe debug2: channel 1: read failed ... debug2: channel 1: input drain -> closed debug2: channel 1: write failed So it's possible that this is not a symptom. Fossil by default doesn't typically enforce permissions over SSH unless something has instructed it to do so. We need to figure out what that is. Is there perhaps some SELinux policy in force that might not like the way the files are being written by Fossil? Also, what version of Fossil is on your remote? Thanks, Andy
(14) By Richard Hipp (drh) on 2025-04-18 15:02:05 in reply to 1.2 [link] [source]
Thanks for working through that problem with us, sebyte. The error message you get back in this scenario has now been improved (as of check-in 2025-04-18T14:59Z) so as to make the problem easier to diagnose and correct should it ever come up again.
(15.3) By sebyte on 2025-04-18 17:13:10 edited from 15.2 in reply to 14 [source]
It is I who should be thanking you Richard (again), and Andy, for your expert help.
Of course, a number of (relatively unimportant) questions remain:
- When you employ option
--ssh-command
, the value is saved in tablerepository.config
and used from then on. Is there a way of reverting to the standard SSH command (other thanDELETE FROM repository.config WHERE name='ssh-command'
)? - Do you have any idea why the
unsent
table was empty while thepush
operation was failing? - As Andy pointed out, line 1421 of file
login.c
tests for value0
. He was using valueoff
and this is why he couldn't reproduce my error to begin with. Shouldn't the help forlocalauth
make this requirement very clear, i.e. "This setting (localauth) must be '0' and only '0'" (or words to that effect), and is it otherwise generally true that0
&off
and1
&on
are interchangeable?
(16) By Richard Hipp (drh) on 2025-04-18 17:29:44 in reply to 15.3 [link] [source]
Your points:
ssh-command is a setting. Adjust it as you would any other setting. For example:
fossil unset ssh-command
The "unsent" table is just an optimization. "fossil push" should work without it. The unsent table normally just saves one round-trip HTTP request.
Fixed by check-in 2025-04-18T15:32Z.
(18) By sebyte on 2025-04-18 18:01:13 in reply to 16 [link] [source]
- Doh!
- Understood (up to a point). That its contents are not a reliable indicator of anything meaningful to an ordinary user is fully understood.
- Great!
(17.1) By Andy Bradford (andybradford) on 2025-04-18 17:52:41 edited from 17.0 in reply to 15.3 [link] [source]
> When you employ option --ssh-command, the value is saved in table > repository.config My apologies. You may not have noticed yesterday that I updated my comment about using --ssh-command to also use the --once option to avoid having the ssh-command setting enabled. Andy
(19) By sebyte on 2025-04-18 18:07:39 in reply to 17.1 [link] [source]
You may not have noticed yesterday that I updated my comment about using --ssh-command to also use the --once option to avoid having the ssh-command setting enabled.
I did notice your update Andy, and I used --once
the first few times I issued the command. I then forgot to use it, probably because I didn't actually know what it's for. I do now! Thanks again for your help.