Cross-compiled fossil.exe does not output stdout in Windows
(1) By smartmic on 2023-09-18 14:06:58 [link] [source]
Hi there,
today I tried to cross-compile Fossil on GNU Linux for usage on Windows1 using these commands:
./configure --with-openssl=none --host=x86_64-w64-mingw32 --with-zlib=/home/smartmic/local/zlib-1.2.13-w32/ --enable-json --static
make
The resulting mingw binary fossil.exe
works without problems on WSL2, however, when I run commands in Windows CMD or Powershell, I see no output, only what is send to stderr.
In other words, this works across WSL2 terminal, Powershell and CMD:
fossil.exe version -v
This is fossil version 2.22 [66ee0beb9b] 2023-05-31 15:26:08 UTC
Compiled on Sep 18 2023 15:47:25 using mingw32 (64-bit)
SQLite 3.42.0 2023-05-16 12:36:15 831d0fb283
zlib 1.2.13, loaded 1.2.13
But this only work in WSL2 for an existing Fossil repository:
fossil.exe status
Any ideas?
- ^ because I need to try the JSON API which the provided binaries do not ship with
(2) By Florian Balmer (florian.balmer) on 2023-09-18 19:48:01 in reply to 1 [source]
What's the output of:
fossil.exe status 1>&2
fossil.exe status | more
fossil.exe status > status.log & type status.log
I'm sorry I'm not familiar with the x86_64-w64-mingw32 cross-platform development environment, and things may also be quite different whether PowerShell and CMD.EXE are run inside a legacy Conhost console or a new Windows Terminal vs. from inside a WSL2 shell (if that is possible at all?).
If x86_64-w64-mingw32 is able to compile code with Win32 API calls, you can build the attached Win32 program to dump standard IO handle information, and run it from CMD.EXE or PowerShell running on a Windows console vs. on WSL2 to investigate the differences at the Win32 process level. Output should be something like this:
>stdio.exe < nul > %TEMP%\sample.log
<STD_INPUT_HANDLE>
[000002bc] Character device
NT object name: \Device\Null
<STD_OUTPUT_HANDLE>
[00000260] File
NT object name: \Device\HarddiskVolume3\Users\Florian\AppData\Local\Temp\sample.log
Win32 file name: \\?\C:\Users\Florian\AppData\Local\Temp\sample.log
<STD_ERROR_HANDLE>
[000001fc] Console or pseudoconsole
If the program doesn't print anything, try the SysInternals DbgView utility to read the system debug output channel.
(The <varargs.h> stuff may need adaptations for non-Microsoft compilers.)
This may help to narrow down the problem, i.e. C runtime library or Fossil vs. Windows to screw up things. But I have to admit I don't really have any good ideas, right now.
========== 8< ========== stdio.c ===============================================
/*
** Dump information about Win32 standard IO handles.
*/
#include <windows.h>
#include <varargs.h>
#define NtDllCall(lpProcName,...) \
(GetProcAddress(GetModuleHandleA("NTDLL"),lpProcName)(__VA_ARGS__))
#define NT_SUCCESS(Status) (((NTSTATUS)(Status)) >= 0)
#define ObjectNameInformation (1)
typedef struct _UNICODE_STRING {
USHORT Length;
USHORT MaximumLength;
PWSTR Buffer;
} UNICODE_STRING, *PUNICODE_STRING;
int FormatStringW(LPWSTR dest,SIZE_T count,LPCWSTR format,...)
{
int ret;
va_list vargs;
va_start(vargs,format);
ret = NtDllCall("_vsnwprintf",dest,count > 0 ? count-1 : 0,format,vargs);
va_end(vargs);
if (count > 0 && dest)
{
if (ret > 0 && ((SIZE_T)ret) <= count-1) dest[ret] = L'\0';
else dest[count-1] = L'\0'; // FIXME: zero dest[0] instead?
}
return ret;
}
void GetIOHandleInfo(
HANDLE hIOHandle,
LPCWSTR pwszDescr,
LPWSTR pwszOutput,
SIZE_T cchOutput
){
DWORD dwConsoleMode;
if (GetConsoleMode(hIOHandle,&dwConsoleMode))
{
FormatStringW(
pwszOutput,
cchOutput,
L"<%s>\n [%08x] Console or pseudoconsole\n",
pwszDescr,
hIOHandle);
}
else
{
NTSTATUS Status;
UCHAR buf[4000] = { 0 }; // FIXME: use dynamic buffer
WCHAR wchFileName[MAX_PATH] = { 0 }; // FIXME: use dynamic buffer
DWORD cchFileName;
DWORD dwFileType = GetFileType(hIOHandle) & ~FILE_TYPE_REMOTE;
switch (dwFileType)
{
case FILE_TYPE_CHAR:
Status = NtDllCall("NtQueryObject",
hIOHandle,ObjectNameInformation,buf,ARRAYSIZE(buf),NULL);
FormatStringW(
pwszOutput,
cchOutput,
L"<%s>\n [%08x] Character device\n NT object name: %s\n",
pwszDescr,
hIOHandle,
NT_SUCCESS(Status) ? ((PUNICODE_STRING)buf)->Buffer : L"n/a");
break;
case FILE_TYPE_DISK:
cchFileName = GetFinalPathNameByHandleW(
hIOHandle,wchFileName,MAX_PATH,FILE_NAME_OPENED);
Status = NtDllCall("NtQueryObject",
hIOHandle,ObjectNameInformation,buf,ARRAYSIZE(buf),NULL);
FormatStringW(
pwszOutput,
cchOutput,
L"<%s>\n [%08x] File\n NT object name: %s\n Win32 file name: %s\n",
pwszDescr,
hIOHandle,
NT_SUCCESS(Status) ? ((PUNICODE_STRING)buf)->Buffer : L"n/a",
cchFileName <= MAX_PATH ? wchFileName : L"n/a");
break;
case FILE_TYPE_PIPE:
FormatStringW(
pwszOutput,
cchOutput,
L"<%s>\n [%08] Pipe\n",
pwszDescr,
hIOHandle);
break;
case FILE_TYPE_UNKNOWN:
FormatStringW(
pwszOutput,
cchOutput,
L"<%s>\n [%08] Unknown\n",
pwszDescr,
hIOHandle);
break;
}
}
}
void main()
{
HANDLE hConsole;
WCHAR wch[1000];
int i;
struct {
HANDLE hIOHandle;
LPCWSTR pwszDescr;
} aSamples[3] = {
{ GetStdHandle(STD_INPUT_HANDLE), L"STD_INPUT_HANDLE" },
{ GetStdHandle(STD_OUTPUT_HANDLE),L"STD_OUTPUT_HANDLE" },
{ GetStdHandle(STD_ERROR_HANDLE), L"STD_ERROR_HANDLE" }
};
hConsole = CreateFileA(
"CONOUT$",
GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_EXISTING,
0,
NULL);
for (i = 0; i < ARRAYSIZE(aSamples); i++)
{
ZeroMemory(wch,ARRAYSIZE(wch)*sizeof(WCHAR));
GetIOHandleInfo(
aSamples[i].hIOHandle,aSamples[i].pwszDescr,wch,ARRAYSIZE(wch));
if (hConsole != INVALID_HANDLE_VALUE)
{
DWORD dwBytesWritten;
WriteConsoleW(hConsole,wch,lstrlenW(wch),&dwBytesWritten,NULL);
}
else
{
OutputDebugStringW(wch);
}
}
CloseHandle(hConsole);
}
(3) By smartmic on 2023-09-19 09:53:27 in reply to 2 [link] [source]
Hi Florian,
thanks for looking into this and for your detailed answer.
Running the first three commands on CMD does not return anything.
The output of the stdio.exe program in CMD is:
>stdio.exe < nul > %TEMP%\sample.log
<STD_INPUT_HANDLE>
[00000058] Character device
NT object name: \Device\Null
<STD_OUTPUT_HANDLE>
[0000005c] File
NT object name: \Device\HarddiskVolume4\Users\smartmic\AppData\Local\Temp\sample.log
Win32 file name: \\?\C:\Users\smartmic\AppData\Local\Temp\sample.log
<STD_ERROR_HANDLE>
[00000060] Console or pseudoconsole
(Yes, <varargs.h>
had to be replaced with <stdarg.h>
for GCC)
This looks the same as in your example. In a WSL2 terminal and running it from Bash, I get
$ ./stdio.exe
<STD_INPUT_HANDLE>
[0000005c] Console or pseudoconsole
<STD_OUTPUT_HANDLE>
[00000060] Console or pseudoconsole
<STD_ERROR_HANDLE>
[00000064] Console or pseudoconsole
I tried also some other things (change codepage to UTF-8, use simple example using fwrite etc.) but have no further idea.
Kind regards, Martin
(4) By Florian Balmer (florian.balmer) on 2023-09-19 16:04:28 in reply to 3 [link] [source]
Running the first three commands on CMD does not return anything.
Ok.
The output of the stdio.exe program in CMD is: ...
I'm sorry the fancy example stdio.exe < nul > %TEMP%\sample.log
was to show
the capabilities of the program. The important thing is that STD_OUTPUT_HANDLE
is the same as STD_ERROR_HANDLE if stdio.exe is run without any redirects (the
same as fossil.exe), as STD_ERROR_HANDLE seems to work. But I think this is also
covered by the fossil.exe status 1>&2
check.
The <stdio.h> library wraps the standard IO handles provided by the OS, and so
the next question is: is the file number/handle underlying stdout
the same as
the working one for stderr
, and does it really point to STD_OUTPUT_HANDLE, or
was is changed for some reason. With MSVC, this can be tested using:
HANDLE hStdout = (HANDLE)_get_osfhandle(_fileno(stdout));
(This test could be added to aSamples[]
in stdio.c, with <stdio.h> included.)
If this is different from the handle returned by querying STD_OUTPUT_HANDLE directly, then the C runtime library for some reason has chosen to use another output handle.
Fossil mostly reuses what <stdio.h> provides, with the expection that UTF-8
output is converted to UTF-16 and sent directly to WriteConsoleW()
, bypassing
the <stdio.h> layer. A failed UTF-8 to UTF-16 conversion could maybe result in
empty output, but this is again unlikely as the STD_ERROR_HANDLE path works.
I tried also ... simple example using fwrite ...
Does this mean something like this done in your x86_64-w64-mingw32 environment doesn't print any output, either:
#include <stdio.h>
main(){
fwrite("Hello, world!",13,1,stdout);
}
This would indicate that the x86_64-w64-mingw32 C runtime library is not working correctly, and the problem is independent of Fossil.
(5) By smartmic on 2023-09-26 14:13:59 in reply to 4 [link] [source]
With MSVC, this can be tested using:
Unfortunately, I do not have MSVC installed and intend also not to do so (too much resources would be needed).
Does this mean something like this done in your x86_64-w64-mingw32 environment doesn't print any output, either:
Well, so along the lines of src/printf.c, I wrote:
#include<stdio.h>
#include<fcntl.h>
#include<io.h>
#include<windows.h>
int main() {
char str[] = "hello world\n";
_setmode(_fileno(stdout), _O_BINARY);
fwrite(str, sizeof(char), sizeof(str) - 1, stdout);
fflush(stdout);
_setmode(_fileno(stdout), _O_TEXT);
return 0;
}
But it prints output fine and I cannot (yet) blame x86_64-w64-mingw32.
The next step would probably diving into the fossil_utf8_to_console
function, but I probably find another way around my issue beforeā¦
(6) By Florian Balmer (florian.balmer) on 2023-09-26 17:47:04 in reply to 5 [link] [source]
Unfortunately, I do not have MSVC installed and intend also not to do so (too much resources would be needed).
Yes, sure, I just meant you could find the x86_64-w64-mingw32 equivalent of
MSVC's _get_osfhandle()
to find out step by step where the <stdio.h>
world
deviates from the Win32 one.
The next step would probably diving into the
fossil_utf8_to_console
function ...
Maybe some _isatty()
problem? Originally, the Cygwin terminal used pipes to
emulate the standard handles (still available through ./mintty --pcon off
), so
_isatty()
had its special heuristics to detect that case (IIRC).
But then again, this probably wouldn't explain the differences between stdout and stderr.
(7) By anonymous on 2024-07-29 12:16:02 in reply to 1 [link] [source]
I see, Oh how I see, big hurty,it really did my my head this path thing, but I found a way, I tried LongPath Tool Program and that sorted it.