Fossil Forum

Assertion failed: (g.json.gc.a && "json_bootstrap_early() was not called!") with --repolist
Login

Assertion failed: (g.json.gc.a && "json_bootstrap_early() was not called!") with --repolist

Assertion failed: (g.json.gc.a && "json_bootstrap_early() was not called!") with --repolist

(1) By Jakob Stoklund Olesen (stoklund) on 2021-01-01 18:29:52 [source]

Hi,

I'm developing a plugin for the buildbot CI framework, and I'd like to use Fossil's JSON API when polling for new commits to test.

The JSON API works great from the command line and when fossil is serving a single repo, but when serving with --repolist, I'm tripping an assertion:

~/Fossils % fossil ui --repolist .
Listening for HTTP requests on TCP port 8080

From another terminal:

~ % curl -i http://localhost:8080/jasmine/json/timeline/checkin
curl: (52) Empty reply from server

The fossil ui process prints:

Assertion failed: (g.json.gc.a && "json_bootstrap_early() was not called!"), function json_page_top, file ./src/json.c, line 2316.

I am seeing the same assertion when running Fossil from inetd.conf like this:

http stream tcp nowait www  /usr/local/bin/fossil fossil http --repolist /mnt/fossils

Accessing the JSON API on that server returns the same assertion:

~ % curl -i http://fossil.local/jasmine/json/timeline/checkin
Assertion failed: (g.json.gc.a && "json_bootstrap_early() was not called!"), function json_page_top, file ./src/json.c, line 2316.

I'm seeing this behavior in two fossil versions:

This is fossil version 2.13 [e7bba4ff36] 2020-11-01 00:13:51 UTC
This is fossil version 2.12.1 [b98ce23d4f] 2020-08-20 13:27:04 UTC

I'm currently working around this issue by using the /timeline.rss entry point instead. This doesn't provide a list of changed files in each commit, though.

(2) By Stephan Beal (stephan) on 2021-01-02 00:52:18 in reply to 1 [link] [source]

Assertion failed: (g.json.gc.a && "json_bootstrap_early() was not called!"), function json_page_top, file ./src/json.c, line 2316.

That assertion message is very helpful - apparently a code path was neglected when we reworked how the JSON bits are bootstrapped sometime early/mid last year.

i'll take a look at this later on today.

(3) By Stephan Beal (stephan) on 2021-01-02 03:02:33 in reply to 1 [link] [source]

I'm currently working around this issue by using the /timeline.rss entry point instead

A better workaround for your case is to use fossil server --localauth instead of fossil ui because...

This problem was resolved, in an ever-so-slightly different form after a report in /forumpost/e4953666d6:

https://fossil-scm.org/fossil/info/dd490d17bec777c4

It was fixed for the server and cgi commands, but not the ui command (which is just a special case of the server command).

Your use case is now fixed in the trunk:

https://fossil-scm.org/fossil/info/e0b51eb2e76b31fe

So ui --repolist ... will work with JSON mode starting with that version.

Please let us know if that does not resolve the problem for you. It "works for me":

# Before fix:
[stephan@nuc:~/fossil/fossil]$ curl -i http://localhost:8080/fossil-skins/json/timeline/checkin?n=2
curl: (52) Empty reply from server


# After:
[stephan@nuc:~/fossil/fossil]$ curl -i http://localhost:8080/fossil-skins/json/timeline/checkin?n=2
HTTP/1.0 200 OK
Date: Sat, 2 Jan 2021 02:52:59 +0000
Connection: close
X-UA-Compatible: IE=edge
Cache-control: no-cache
X-Frame-Options: SAMEORIGIN
Content-Type: application/json; charset=utf-8; charset=utf-8
Content-Length: 3722

{"fossil":"67d79d23e1929ee60c6b0668 ...

(Hmmmm. charset is getting duplicated on the Content-Type header.)

(4) By Jakob Stoklund Olesen (stoklund) on 2021-01-02 05:20:54 in reply to 3 [link] [source]

Thanks for looking at this so quickly! I'll try and build from sources tomorrow.

BTW, did you also check fossil http --repolist? That's my real use case as it happens.

(5) By Stephan Beal (stephan) on 2021-01-02 05:26:38 in reply to 4 [link] [source]

BTW, did you also check fossil http --repolist? That's my real use case as it happens.

i had not considered that one, but just checked that addition in. That command is not really intended for use by clients - it's an internal thing which is used by, e.g., ssh connections.

(6) By Jakob Stoklund Olesen (stoklund) on 2021-01-02 05:39:27 in reply to 5 [link] [source]

Thanks! I must have gotten that command from the "running under inetd" documentation.

From looking at the function you fixed, I suspect that there may also be trouble with fossil all server and fossil all ui. I don't have a need for these commands, though, and certainly not their JSON API.

(7) By Stephan Beal (stephan) on 2021-01-02 05:48:34 in reply to 6 [link] [source]

I must have gotten that command from the "running under inetd" documentation.

It's not outright forbidden, it's just not "really" there for general client use.

From looking at the function you fixed, I suspect that there may also be trouble with fossil all server and fossil all ui.

fossil all launches a new instance of fossil for each repository, stripping off the "all" part of the command, so those should be okay:

https://fossil-scm.org/fossil/file?ci=9d53f2f6a22037e4&name=src%2Fallrepo.c&ln=424-425

(9) By Andy Bradford (andybradford) on 2021-01-02 16:29:33 in reply to 6 [link] [source]

> I  must  have gotten  that  command  from  the "running  under  inetd"
> documentation.

And this  is correct. "fossil http"  is used by inetd  configurations et
al. I have mine setup with tcpserver and daemontools like this:

#!/bin/sh
exec 2>&1
exec softlimit -m20000000 tcpserver \
  -vDRHl0 192.168.2.3 1951 chroot -u _fossil /var/fossil /usr/local/sbin/stunnel /etc/fossil/ssl.conf

And in my ssl.conf:

# cat ssl.conf 
#verify = 3
verify = 1
CApath = /etc/fossil/certs
cert = /etc/fossil/ssl.pem
exec = /usr/local/bin/fossil
execargs = fossil http --https /fossil-ssl
foreground = yes
syslog = no
debug = 4

Notice that execargs calls "fossil http" here which is precisely what is
needed for this kind of configuration.

Andy

(8.3) By Warren Young (wyoung) on 2021-01-02 15:51:11 edited from 8.2 in reply to 5 [link] [source]

it's an internal thing which is used by, e.g., ssh connections.

For ssh:// URLs at least, you're thinking of test-http.

When you want Fossil's RBAC system to come into effect, as it does not with test-http, then you want fossil http, as in socket activation schemes like my macOS launchd and Linux systemd configs. I got that from drh's inetd config which uses this method for the same reason I did in those articles. I expect the OP got it from the same place.

So yeah, the JSON API does legitimately need to work for fossil http.

(10) By Jakob Stoklund Olesen (stoklund) on 2021-01-02 16:57:42 in reply to 5 [link] [source]

Okay, now I've checked out the trunk sources which gave me Fossil 2.14 [49f68be83b] 2021-01-02 13:39:46. I tested the JSON API with --repolist, and I can confirm that it works with:

fossil server
fossil ui
fossil http

Thanks again for fixing this so quickly!

While I was at it, I also tested the JSON API with these commands:

fossil all server --localauth
fossil all ui

These are still tripping the same assertion, both with the released 2.13 and the newly built version with the fix. Again, I don't have a use for the JSON API with these commands, so this is not a problem for me.

(11) By Stephan Beal (stephan) on 2021-01-02 17:21:47 in reply to 10 [link] [source]

These are still tripping the same assertion, both with the released 2.13 and the newly built version with the fix. Again, I don't have a use for the JSON API with these commands, so this is not a problem for me.

That's unexpected. It would be impossible to make reliable use of the JSON API with those commands because of the unspecified execution order leading to unpredictable port numbers, so that's a low-priority bug but it's a bug nonetheless.

@Warren: indeed, i confused it with test-http. Thank you for the correction.

(12) By Jakob Stoklund Olesen (stoklund) on 2021-01-02 18:09:12 in reply to 11 [link] [source]

I wonder if you are remembering a previous version of these commands?

When I run fossil all ui, I get a single directory page at localhost:8080 containing links like these:

<a href='/Users/jolesen/Fossils/config/home' target='_blank'>/Users/jolesen/Fossils/config.fossil</a>
<a href='/Users/jolesen/Fossils/fossil/home' target='_blank'>/Users/jolesen/Fossils/fossil.fossil</a>

Each repo is served from port 8080, but with the full absolute path to the repo in the URL, for example http://localhost:8080/Users/jolesen/Fossils/config/timeline. The regex in json_request_is_json_api() only strips one leading directory from the URL path, which is why I suspected trouble. (I don't really understand how any of this code works, though).

(13) By Stephan Beal (stephan) on 2021-01-02 18:42:22 in reply to 12 [link] [source]

I wonder if you are remembering a previous version of these commands?

i've never once (in 13 years) had a user for/used "all" - i was working only from a cursory glance at the code. Indeed, ui/server are handled as special cases there. Sigh.

The regex in json_request_is_json_api() only strips one leading directory from the URL path, which is why I suspected trouble. (I don't really understand how any of this code works, though).

The resulting path for those can be arbitrarily long, meaning the easy fix for it is to have the regex assume that any number of directories before /json are legal, but that's really only the case for "all server/ui". Hmmm. That level of laxness in the regex doesn't sit well with me, though, so i'll need to mull it over before fixing that.

Thank you for point that out.

(14) By Jakob Stoklund Olesen (stoklund) on 2021-01-02 19:56:59 in reply to 13 [link] [source]

That level of laxness in the regex doesn't sit well with me, though, so i'll need to mull it over before fixing that.

I agree. Someone named "Jimmy Son" might keep all their repos in /home/json/fossils/*.fossil. They would see HTML URLs like http://localhost:8080/home/json/config/timeline when running fossil all ui.

(15.1) By S. Ross Gohlke (rossgohlke) on 2023-02-17 16:30:39 edited from 15.0 in reply to 13 [link] [source]

Did anyone ever mull a solution?

I have a collection or repositories with a hierarchical depth of up to 6 directories and it would be nice to access /json on all of them, not just those at the top level. They are being served by a CGI script running under lighttpd, and everything besides /json works fine. I have also tested with "fossil server". In all cases, I am getting the same error.

Assertion failed: (g.json.gc.a && "json_bootstrap_early() was not called!"), function json_page_top, file ./src/json.c, line 2311.

The funny thing is, I could swear this worked in an earlier incarnation.

Also, I never knew about "fossil all server", thanks for that tip.

(16) By Stephan Beal (stephan) on 2023-02-17 16:30:11 in reply to 15.0 [link] [source]

Did anyone ever mull a solution?

Nope, following the mantra of "it's not a problem until it's a problem." ;)

I have a collection or repositories with a hierarchical depth of up to 6 directories

My gut-instinct reactions to that are (A) "don't do that" and (B) use one CGI script per repository (where it's not a problem).

and it would be nice to access /json on all of them, not just those at the top level.

Unfortunately, the multi-repository approach to serving, in particular with directory depths greater than 1, is "simply problematic" with regards to the determination of whether or not the current path belongs to the JSON API. This is a problem because, for purposes of handling the GET/POST input, we have to know early on whether or not this is a JSON-centric request, and that handling has to be done before the dispatcher has determined, with certainty, that this is a /json path. Any attempts made up until that point are guesses based on the full path, and that breaks down when one fossil instance serves multiple repositories. It's possible for either end of a given path to contain the string "/json", either before or after a "real" "/json" path: /foo/json/myrepo/json/foo/json. Until the dispatcher has done it's work, we don't reliably know for sure which of those 3 /json parts (if any!) are intended to point to a /json API endpoint.

i can't currently offer a reliable solution to this beyond using a separate CGI script per repository. When served that way, all path resolution is unambiguous.

(17.1) By S. Ross Gohlke (rossgohlke) on 2023-02-17 16:48:00 edited from 17.0 in reply to 16 [link] [source]

Thanks for the detailed response.

This is a problem because, for purposes of handling the GET/POST input, we have to know early on whether or not this is a JSON-centric request

I guess this explains why non-json requests might work where json requests do not, because of how early the json handling branches.

using a separate CGI script per repository. When served that way, all path resolution is unambiguous.

As I understand this thread, "fossil all server" was investigated and found lacking as a shortcut to the script-per-repository approach.

(18) By Stephan Beal (stephan) on 2023-02-17 16:55:50 in reply to 17.1 [link] [source]

I guess this explains why non-json requests might work where json requests do not, because of how early the json handling branches.

Exactly. If we knew for certain, at the time the GET/POST bits are handled, how the dispatcher would answer, it wouldn't be an issue. It's an order-of-operations problem with no straightforward fix.

As I understand this thread, "fossil all server" was investigated and found lacking as a shortcut to the script-per-repository approach.

AFAIK, "all server" is going to have a similar limitation, but i've never been a fan of the multiple-repos-per-instance serving modes and don't use them so cannot say for certain off hand.