Hi,

I'm running Fossil with CGI and Apache on my Ubuntu server, which automatically updates and rebuilds Fossil every Wednesday morning. This morning, I got an error report that my Fossil site was down; every request gave the same error:

Bad Request: missing SCRIPT_NAME

My server was running check-in 3f8cdaa1. After checking out the timeline, I thought the description of c06e0b2d "Changes to support CGI on IIS web servers" sounded like a likely culprit, so I reverted to the preceding checkin c6dfb558, and now my server works fine again.

I don't have the time to delve into it further at the moment, but I thought people might like to know about this.

i've just installed trunk 3f8cdaa15e on my hoster, also Apache+CGI, and cannot reproduce this, but...

Your case seems to be related to this change from c06e0b2b:

https://fossil-scm.org/fossil/artifact?ln=1196&name=e36b8b77f5ff8431

That changes the behaviour of cgi_parameter() to treat variables which are set but have no value the same way as variables which are unset. If your local SCRIPT_NAME is empty, it would trigger this condition and, subsequently, trigger the missing SCRIPT_NAME case.

What might cause your SCRIPT_NAME to be empty... no idea. Can you confirm, under /test_env (requires admin login), whether SCRIPT_NAME is set, or is it failing before you can even get to that point?

My hoster is currently running 3f8cdaa15e, but there have been (judging solely by commit messages) no changes made since c06e0 which would affect what you're seeing. i.e., if you update to 3f8cda i expect you'll still be seeing the same problem. (But why is unclear.)

When /test_env is run from my site's top-most index CGI script, it reports SCRIPT_NAME = /index. Have you somehow managed to host a top-level CGI script without a name, such that hitting / on your site runs a CGI script without redirecting the user to that script (e.g., so the timeline is at /timeline instead of /the-script/timeline)? If so, (A) that would seemingly explain the behaviour and (B) i'd very much love to know how you did it.

I tested /test_env under checkin 45953a4ad0 on a test repository, but I merely got the Bad Request: missing SCRIPT_NAME error.

When I tried the same under the older checkin c6dfb55802, that reports the following:

DOCUMENT_ROOT = /***/fossil/www/
GATEWAY_INTERFACE = CGI/1.1
HTTPS = on
(...)
HTTP_HOST = fossil.2of4.net
HTTP_REFERER = https://fossil.2of4.net/Test/login?g=/Test/test_env
(...)
PATH_INFO = test_env
PATH_TRANSLATED = /***/cgi-bin/fossil.2of4.net/Test/test_env
QUERY_STRING =
REMOTE_ADDR = ***
REMOTE_PORT = 62945
REQUEST_METHOD = GET
REQUEST_URI = /Test/test_env
SCRIPT_FILENAME = /***/cgi-bin/fossil.2of4.net
SCRIPT_NAME = /Test
SERVER_PROTOCOL = HTTP/1.1 

(I've masked out some irrelevant stuff with ***.)

You'll notice that SCRIPT_NAME is not reported as empty at all...

i am unfortunately at a complete loss to explain the error. It's running as expected on my local system (Mint Linux) and hoster (some unspecified linux), and the code path which leads to the error you're seeing is triggered only if SCRIPT_NAME is empty or unset.

There seems to be a workaround, but it's not pretty: in your CGI wrapper script, add this line:

setenv: SCRIPT_NAME /repo/cgi/uri

Being sure to include the leading slash.

In my tests that will supplant the SCRIPT_NAME provided by Apache (set it to a bogus path and open your browser's dev tools to see it in action).

Presumably repolist is the culprit there. i've never used it, instead hosting one cgi per repo. In such cases, injecting the script name directly after the domain name is exactly what's needed.

As a debugging step, please temporarily replace the CGI with a script that is something like this:

#!/bin/sh
echo 'Content-Type: text/plain'
echo
env | sort

The post the output.

With the latest check-in, you can now put a line in the CGI control file like this:

   cgi-debug: FILENAME

This will cause CGI-related debugging information to be appended to FILENAME. If you will kindly rebuild using the latest trunk check-in of Fossil, set that option (temporarily, of course, it generates a lot of output and probably is not suitable for production) and then examine the environment dump that appears at the beginning of each request log, that might help to clear up this problem.

Good point. Here's the output:

CONTEXT_DOCUMENT_ROOT=/var/www/fossil/www/
CONTEXT_PREFIX=
DOCUMENT_ROOT=/var/www/fossil/www/
GATEWAY_INTERFACE=CGI/1.1
HTTPS=on
HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
HTTP_ACCEPT_ENCODING=gzip, deflate, br
HTTP_ACCEPT_LANGUAGE=nl,en-GB;q=0.8,en;q=0.7,fr;q=0.5,de;q=0.3,af;q=0.2
HTTP_CONNECTION=keep-alive
HTTP_COOKIE=fossil-934d600d98d49970=30F0F2F83CE50F15AA5CC795140FABD1534E08667349FF4AAB%2F934d600d98d49970%2Ftinus
HTTP_DNT=1
HTTP_HOST=fossil.2of4.net
HTTP_UPGRADE_INSECURE_REQUESTS=1
HTTP_USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
PATH_INFO=/Test
PATH_TRANSLATED=/<redacted-path-to>/cgi-bin/fossil.2of4.net/Test
PWD=/<redacted-path-to>/cgi-bin
QUERY_STRING=
REMOTE_ADDR=<redacted>
REMOTE_PORT=53624
REQUEST_METHOD=GET
REQUEST_SCHEME=https
REQUEST_URI=/Test
SCRIPT_FILENAME=/<redacted-path-to>/cgi-bin/fossil.2of4.net
SCRIPT_NAME=
SCRIPT_URI=https://fossil.2of4.net/Test
SCRIPT_URL=/Test
SERVER_ADDR=185.56.144.177
SERVER_ADMIN=fossil.2of4.net@<redacted>
SERVER_NAME=fossil.2of4.net
SERVER_PORT=443
SERVER_PROTOCOL=HTTP/1.1
SERVER_SIGNATURE=<address>Apache/2.4.29 (Ubuntu) Server at fossil.2of4.net Port 443</address>
SERVER_SOFTWARE=Apache/2.4.29 (Ubuntu)
SSL_TLS_SNI=fossil.2of4.net

(The /<redacted-path-to>/cgi-bin/ is identical throughout).

So apparently here, SCRIPT_NAME is empty.

Thank you for that, I will try this as soon as I can find some time, possibly tonight or tomorrow evening.

Notice that SCRIPT_URI a d SCRIPT_URL seem to be swapped. That URI is what i would expect to see in SCRIPT_NAME.

So, I recommend adding a line to the CGI file like this:

  setenv: SCRIPT_NAME /Test

Do this, and remove any "cgi-debug:" line that you might be using, and then I think it should probably work.

The larger issue is whether this is a local configuration problem on a single Apache server, or if there is a larger and more pervasive problem with Apache not setting SCRIPT_NAME correctly. If Apache has started substituting SCRIPT_URI in place of SCRIPT_NAME then perhaps we should enhance Fossil to look for SCRIPT_URI if SCRIPT_NAME is unavailable.

Thanks!

I've added the cgi-debug line, built the latest version, ran my first test request https://fossil.2of4.net/Test, which returned the same error "Bad request: missing SCRIPT_NAME"; and saw that a second request came in, for https://fossil.2of4.net/npp_preview/zip/PreviewHTML32.zip?name=&uuid=v1.3.2.0-32.

Here's the cgi-debug output for these two requests:

-------- BEGIN cgi at 2019-11-12 20:02:54 --------
DOCUMENT_ROOT = /var/fossil/www/
GATEWAY_INTERFACE = CGI/1.1
HTTP_ACCEPT = text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
HTTP_ACCEPT_ENCODING = gzip, deflate, br
HTTP_ACCEPT_LANGUAGE = nl,en-GB;q=0.8,en;q=0.7,fr;q=0.5,de;q=0.3,af;q=0.2
HTTP_CONNECTION = keep-alive
HTTP_HOST = fossil.2of4.net
HTTP_USER_AGENT = Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0
PATH_INFO = /Test/index
PATH_TRANSLATED = /usr/lib/cgi-bin/fossil.2of4.net/Test/index
REMOTE_ADDR = 85.149.53.206
REMOTE_PORT = 56985
REQUEST_METHOD = GET
REQUEST_URI = /Test/index
SCRIPT_FILENAME = /usr/lib/cgi-bin/fossil.2of4.net
SERVER_PROTOCOL = HTTP/1.1
mem-match [REQUEST_URI] = [/Test/index]
no-match [SCRIPT_NAME]
mem-match [PATH_INFO] = [/Test/index]
no-match [no_json]
mem-match [HTTP_ACCEPT_ENCODING] = [gzip, deflate, br]
mem-match [REQUEST_METHOD] = [GET]
-------- END cgi ---------
-------- BEGIN cgi at 2019-11-12 20:03:32 --------
DOCUMENT_ROOT = /var/fossil/www/
GATEWAY_INTERFACE = CGI/1.1
HTTP_ACCEPT = */*
HTTP_HOST = fossil.2of4.net
HTTP_USER_AGENT = WinGup/
PATH_INFO = /npp_preview/zip/PreviewHTML32.zip
PATH_TRANSLATED = /usr/lib/cgi-bin/fossil.2of4.net/npp_preview/zip/PreviewHTML32.zip
QUERY_STRING = name=&uuid=v1.3.2.0-32
REMOTE_ADDR = 189.115.81.56
REMOTE_PORT = 50143
REQUEST_METHOD = GET
REQUEST_URI = /npp_preview/zip/PreviewHTML32.zip%3Fname%3D%26uuid%3Dv1.3.2.0-32
SCRIPT_FILENAME = /usr/lib/cgi-bin/fossil.2of4.net
SCRIPT_NAME = /npp_preview/zip/PreviewHTML32.zip?name=&uuid=v1.3.2.0-32
SERVER_PROTOCOL = HTTP/1.1
mem-match [REQUEST_URI] = [/npp_preview/zip/PreviewHTML32.zip%3Fname%3D%26uuid%3Dv1.3.2.0-32]
mem-match [SCRIPT_NAME] = [/npp_preview/zip/PreviewHTML32.zip?name=&uuid=v1.3.2.0-32]
mem-match [PATH_INFO] = [/npp_preview/zip/PreviewHTML32.zip]
no-match [no_json]
no-match [HTTP_COOKIE]
mem-match [QUERY_STRING] = [name=&uuid=v1.3.2.0-32]
mem-match [REMOTE_ADDR] = [189.115.81.56]
no-match [CONTENT_LENGTH]
no-match [CONTENT_TYPE]
mem-match [PATH_INFO] = [/npp_preview/zip/PreviewHTML32.zip]
no-match [utc]
no-match [localtime]
mem-match [SCRIPT_NAME] = [/npp_preview/zip/PreviewHTML32.zip?name=&uuid=v1.3.2.0-32]
mem-match [HTTP_HOST] = [fossil.2of4.net]
env-match [HTTPS] = [on]
mem-match [SCRIPT_NAME] = [/npp_preview/zip/PreviewHTML32.zip?name=&uuid=v1.3.2.0-32/npp_preview]
mem-match [HTTPS] = [on]
mem-match [REMOTE_ADDR] = [189.115.81.56]
no-match [fossil-9cfd0b89039e0640]
no-match [REMOTE_USER]
no-match [isrobot]
mem-match [HTTP_USER_AGENT] = [WinGup/]
mem-match [name] = []
no-match [r]
mem-match [uuid] = [v1.3.2.0-32]
no-match [in]
no-match [ex]
no-match [HTTP_IF_NONE_MATCH]
no-match [debug]
no-match [HTTP_REFERER]
no-match [HTTP_ACCEPT_ENCODING]
mem-match [REQUEST_METHOD] = [GET]
-------- END cgi ---------

Since I saw that the previous version apparently supported a similar debug keyword in the CGI control file, I added that, restored a 'working' binary of fossil, and requested the same two repositories's home pages (https://fossil.2of4.net/Test and https://fossil.2of4.net/npp_preview):

env-match [REQUEST_URI] = [/Test/index]
env-match [SCRIPT_NAME] = []
env-match [PATH_INFO] = [/Test/index]
no-match [no_json]
env-match [HTTP_COOKIE] = [fossil-934d600d98d49970=30F0F2F83CE50F15AA5CC795140FABD1534E08667349FF4AAB%2F934d600d98d49970%2Ftinus]
env-match [QUERY_STRING] = []
env-match [REMOTE_ADDR] = [85.149.53.206]
no-match [CONTENT_LENGTH]
no-match [CONTENT_TYPE]
mem-match [PATH_INFO] = [/Test/index]
no-match [utc]
no-match [localtime]
mem-match [SCRIPT_NAME] = []
env-match [HTTP_HOST] = [fossil.2of4.net]
env-match [HTTPS] = [on]
mem-match [SCRIPT_NAME] = [/Test]
mem-match [REMOTE_ADDR] = [85.149.53.206]
mem-match [fossil-934d600d98d49970] = [30F0F2F83CE50F15AA5CC795140FABD1534E08667349FF4AAB/934d600d98d49970/tinus]
no-match [isrobot]
mem-match [name] = [Test Fossil Project]
no-match [id]
mem-match [HTTPS] = [on]
no-match [showqp]
env-match [HTTP_ACCEPT_ENCODING] = [gzip, deflate, br]
env-match [REQUEST_METHOD] = [GET]
DONE
env-match [REQUEST_URI] = [/Test/style.css?id=73500f26]
env-match [SCRIPT_NAME] = []
env-match [PATH_INFO] = [/Test/style.css]
no-match [no_json]
env-match [HTTP_COOKIE] = [fossil-934d600d98d49970=30F0F2F83CE50F15AA5CC795140FABD1534E08667349FF4AAB%2F934d600d98d49970%2Ftinus]
env-match [QUERY_STRING] = [id=73500f26]
env-match [REMOTE_ADDR] = [85.149.53.206]
no-match [CONTENT_LENGTH]
no-match [CONTENT_TYPE]
mem-match [PATH_INFO] = [/Test/style.css]
no-match [utc]
no-match [localtime]
mem-match [SCRIPT_NAME] = []
env-match [HTTP_HOST] = [fossil.2of4.net]
env-match [HTTPS] = [on]
mem-match [SCRIPT_NAME] = [/Test]
mem-match [HTTPS] = [on]
env-match [HTTP_ACCEPT_ENCODING] = [gzip, deflate, br]
env-match [REQUEST_METHOD] = [GET]
DONE
env-match [REQUEST_URI] = [/npp_preview/home]
env-match [SCRIPT_NAME] = []
env-match [PATH_INFO] = [/npp_preview/home]
no-match [no_json]
env-match [HTTP_COOKIE] = [fossil-9cfd0b89039e0640=16359387067AAFA1B515F8678DA130E6215826BAFDF73F5A10%2F9cfd0b89039e0640%2Ftinus; fossil_display_settings=n%3D50%2Css%3Dm%2Cadvm%3D1%2Cy%3Dci]
env-match [QUERY_STRING] = []
env-match [REMOTE_ADDR] = [85.149.53.206]
no-match [CONTENT_LENGTH]
no-match [CONTENT_TYPE]
mem-match [PATH_INFO] = [/npp_preview/home]
no-match [utc]
no-match [localtime]
mem-match [SCRIPT_NAME] = []
env-match [HTTP_HOST] = [fossil.2of4.net]
env-match [HTTPS] = [on]
mem-match [SCRIPT_NAME] = [/npp_preview]
mem-match [HTTPS] = [on]
mem-match [REMOTE_ADDR] = [85.149.53.206]
mem-match [fossil-9cfd0b89039e0640] = [16359387067AAFA1B515F8678DA130E6215826BAFDF73F5A10/9cfd0b89039e0640/tinus]
no-match [isrobot]
mem-match [PATH_INFO] = [home]
mem-match [name] = [Preview HTML]
no-match [id]
mem-match [HTTPS] = [on]
no-match [showqp]
env-match [HTTP_ACCEPT_ENCODING] = [gzip, deflate, br]
env-match [REQUEST_METHOD] = [GET]
DONE

In that version, the first mem-match [SCRIPT_NAME] = [] is always followed by a second (correct) mem-match [SCRIPT_NAME] = [/<repo>], where <repo> stands for the repository name; in this case, Test and npp_preview, respectively.

I'm wondering why the newer build(s) of fossil no longer produce this result?

Alas, that line would work if I only had the one repo named Test; but not when I'm using repolist to serve any one of some 50 repos.

Also, as I mentioned in my reply to Stephan, anything I put after setenv: SCRIPT_NAME gets inserted between the domain name and the repository name.

But I'll see what I can find out about the Apache configuration for that site; it may well be an outlandish configuration. I set it up many years ago with little to no prior knowledge of Apache, CGI (or Fossil, for that matter), tinkering until it worked. The only thing I've (had) checked since then, was for possible security leaks.

Here's the cgi-debug output for these two requests:

i HAVE to know how you're getting that to work, because i've spent literally the past 2 hours pulling out my hair and littering the sources with assert() and fprintf() calls and i can get no output to go to that file. On my system, i have:

cgi-debug: /tmp/cgi-debug

And that file is:

-rw-rw-rw- 1 www-data www-data 0 Dec 13 20:33 /tmp/cgi-debug

And the /tmp directory is world-writable.

This is a CGI and there is no chroot() going on.

That file is being opened in main.c, which i can prove with asserts() and fprintf(stderr...), which goes to apache's system-wide error log. Even so, nothing i write there (using cgi_debug()) ever ends up in that file, despite the file being flush()ed after ever write.

stderr-level debug output is telling me:

// When opening the file:
output file=0x5570c0112370 /tmp/cgi-debug

// After cgi_debug() calls vfprintf():
cgi_debug() errno=0, rc=51, g.fDebug=0x5570c0112370

The only way i'm able to produce cgi_debug() output is to explicitly hard-code the g.fDebug FILE handle to stderr. i am at a complete, utter loss as to why the handle, which is being correctly opened and written to is getting no data. It's mtime is also not changing.

i'm pulling my hair out trying to figure this out.

Any tips?

Sorry, I’ve had a busy weekend, and did not see your reply.

I’m afraid I’m not going to be of much help...

All I did was

1. build fossil from commit [4a7760e368]
2. add the line cgi-debug: /var/fossil/config/cgi-debug.txt to the CGI control file (the cgi-debug.txt file did not yet exist, but is located in the same directory the errorlog is written to).

and it just started writing output on the next request.

I still haven’t looked into it very thoroughly (what with a heavy bout of flu affecting the entire family, and the holiday season), but the Apache configuration looks rather convoluted, making heavy use of RewriteEngine.

I'll keep using the older build of Fossil for now, and look into reconfiguring Apache, so that it populates the SCRIPT_NAME variable properly, when I get the time.

Thanks for the debugging help, and happy end-of-year celebrations to all!