Fossil Forum

Cross-compiled fossil.exe does not output stdout in Windows
Login

Cross-compiled fossil.exe does not output stdout in Windows

Cross-compiled fossil.exe does not output stdout in Windows

(1) By smartmic on 2023-09-18 14:06:58 [link] [source]

Hi there,

today I tried to cross-compile Fossil on GNU Linux for usage on Windows1 using these commands:

./configure --with-openssl=none --host=x86_64-w64-mingw32 --with-zlib=/home/smartmic/local/zlib-1.2.13-w32/ --enable-json --static
make

The resulting mingw binary fossil.exe works without problems on WSL2, however, when I run commands in Windows CMD or Powershell, I see no output, only what is send to stderr.

In other words, this works across WSL2 terminal, Powershell and CMD:

fossil.exe version -v
This is fossil version 2.22 [66ee0beb9b] 2023-05-31 15:26:08 UTC
Compiled on Sep 18 2023 15:47:25 using mingw32 (64-bit)
SQLite 3.42.0 2023-05-16 12:36:15 831d0fb283
zlib 1.2.13, loaded 1.2.13

But this only work in WSL2 for an existing Fossil repository:

fossil.exe status

Any ideas?


  1. ^ because I need to try the JSON API which the provided binaries do not ship with

(2) By Florian Balmer (florian.balmer) on 2023-09-18 19:48:01 in reply to 1 [source]

What's the output of:

fossil.exe status 1>&2
fossil.exe status | more
fossil.exe status > status.log & type status.log

I'm sorry I'm not familiar with the x86_64-w64-mingw32 cross-platform development environment, and things may also be quite different whether PowerShell and CMD.EXE are run inside a legacy Conhost console or a new Windows Terminal vs. from inside a WSL2 shell (if that is possible at all?).

If x86_64-w64-mingw32 is able to compile code with Win32 API calls, you can build the attached Win32 program to dump standard IO handle information, and run it from CMD.EXE or PowerShell running on a Windows console vs. on WSL2 to investigate the differences at the Win32 process level. Output should be something like this:

>stdio.exe < nul > %TEMP%\sample.log
<STD_INPUT_HANDLE>
 [000002bc] Character device
 NT object name: \Device\Null
<STD_OUTPUT_HANDLE>
 [00000260] File
 NT object name: \Device\HarddiskVolume3\Users\Florian\AppData\Local\Temp\sample.log
 Win32 file name: \\?\C:\Users\Florian\AppData\Local\Temp\sample.log
<STD_ERROR_HANDLE>
 [000001fc] Console or pseudoconsole

If the program doesn't print anything, try the SysInternals DbgView utility to read the system debug output channel.

(The <varargs.h> stuff may need adaptations for non-Microsoft compilers.)

This may help to narrow down the problem, i.e. C runtime library or Fossil vs. Windows to screw up things. But I have to admit I don't really have any good ideas, right now.

========== 8< ========== stdio.c ===============================================
/*
** Dump information about Win32 standard IO handles.
*/

#include <windows.h>
#include <varargs.h>

#define NtDllCall(lpProcName,...) \
  (GetProcAddress(GetModuleHandleA("NTDLL"),lpProcName)(__VA_ARGS__))

#define NT_SUCCESS(Status) (((NTSTATUS)(Status)) >= 0)

#define ObjectNameInformation (1)

typedef struct _UNICODE_STRING {
    USHORT Length;
    USHORT MaximumLength;
    PWSTR  Buffer;
} UNICODE_STRING, *PUNICODE_STRING;

int FormatStringW(LPWSTR dest,SIZE_T count,LPCWSTR format,...)
{
  int ret;
  va_list vargs;
  va_start(vargs,format);
  ret = NtDllCall("_vsnwprintf",dest,count > 0 ? count-1 : 0,format,vargs);
  va_end(vargs);
  if (count > 0 && dest)
  {
    if (ret > 0 && ((SIZE_T)ret) <= count-1) dest[ret] = L'\0';
    else dest[count-1] = L'\0';   // FIXME: zero dest[0] instead?
  }
  return ret;
}

void GetIOHandleInfo(
  HANDLE  hIOHandle,
  LPCWSTR pwszDescr,
  LPWSTR  pwszOutput,
  SIZE_T  cchOutput
){
  DWORD dwConsoleMode;
  if (GetConsoleMode(hIOHandle,&dwConsoleMode))
  {
    FormatStringW(
      pwszOutput,
      cchOutput,
      L"<%s>\n [%08x] Console or pseudoconsole\n",
      pwszDescr,
      hIOHandle);
  }
  else
  {
    NTSTATUS Status;
    UCHAR buf[4000] = { 0 };              // FIXME: use dynamic buffer
    WCHAR wchFileName[MAX_PATH] = { 0 };  // FIXME: use dynamic buffer
    DWORD cchFileName;
    DWORD dwFileType = GetFileType(hIOHandle) & ~FILE_TYPE_REMOTE;
    switch (dwFileType)
    {
      case FILE_TYPE_CHAR:
        Status = NtDllCall("NtQueryObject",
                   hIOHandle,ObjectNameInformation,buf,ARRAYSIZE(buf),NULL);
        FormatStringW(
          pwszOutput,
          cchOutput,
          L"<%s>\n [%08x] Character device\n NT object name: %s\n",
          pwszDescr,
          hIOHandle,
          NT_SUCCESS(Status) ? ((PUNICODE_STRING)buf)->Buffer : L"n/a");
          break;
      case FILE_TYPE_DISK:
        cchFileName = GetFinalPathNameByHandleW(
                        hIOHandle,wchFileName,MAX_PATH,FILE_NAME_OPENED);
        Status = NtDllCall("NtQueryObject",
                   hIOHandle,ObjectNameInformation,buf,ARRAYSIZE(buf),NULL);
        FormatStringW(
          pwszOutput,
          cchOutput,
          L"<%s>\n [%08x] File\n NT object name: %s\n Win32 file name: %s\n",
          pwszDescr,
          hIOHandle,
          NT_SUCCESS(Status) ? ((PUNICODE_STRING)buf)->Buffer : L"n/a",
          cchFileName <= MAX_PATH ? wchFileName : L"n/a");
        break;
      case FILE_TYPE_PIPE:
        FormatStringW(
          pwszOutput,
          cchOutput,
          L"<%s>\n [%08] Pipe\n",
          pwszDescr,
          hIOHandle);
        break;
      case FILE_TYPE_UNKNOWN:
        FormatStringW(
          pwszOutput,
          cchOutput,
          L"<%s>\n [%08] Unknown\n",
          pwszDescr,
          hIOHandle);
        break;
    }
  }
}

void main()
{
  HANDLE hConsole;
  WCHAR wch[1000];
  int i;
  struct {
    HANDLE hIOHandle;
    LPCWSTR pwszDescr;
  } aSamples[3] = {
    { GetStdHandle(STD_INPUT_HANDLE), L"STD_INPUT_HANDLE" },
    { GetStdHandle(STD_OUTPUT_HANDLE),L"STD_OUTPUT_HANDLE" },
    { GetStdHandle(STD_ERROR_HANDLE), L"STD_ERROR_HANDLE" }
  };
  hConsole = CreateFileA(
                "CONOUT$",
                GENERIC_WRITE,
                FILE_SHARE_READ | FILE_SHARE_WRITE,
                NULL,
                OPEN_EXISTING,
                0,
                NULL);
  for (i = 0; i < ARRAYSIZE(aSamples); i++)
  {
    ZeroMemory(wch,ARRAYSIZE(wch)*sizeof(WCHAR));
    GetIOHandleInfo(
      aSamples[i].hIOHandle,aSamples[i].pwszDescr,wch,ARRAYSIZE(wch));
    if (hConsole != INVALID_HANDLE_VALUE)
    {
      DWORD dwBytesWritten;
      WriteConsoleW(hConsole,wch,lstrlenW(wch),&dwBytesWritten,NULL);
    }
    else
    {
      OutputDebugStringW(wch);
    }
  }
  CloseHandle(hConsole);
}

(3) By smartmic on 2023-09-19 09:53:27 in reply to 2 [link] [source]

Hi Florian,

thanks for looking into this and for your detailed answer.

Running the first three commands on CMD does not return anything.

The output of the stdio.exe program in CMD is:

>stdio.exe < nul > %TEMP%\sample.log
<STD_INPUT_HANDLE>
 [00000058] Character device
 NT object name: \Device\Null
<STD_OUTPUT_HANDLE>
 [0000005c] File
 NT object name: \Device\HarddiskVolume4\Users\smartmic\AppData\Local\Temp\sample.log
 Win32 file name: \\?\C:\Users\smartmic\AppData\Local\Temp\sample.log
<STD_ERROR_HANDLE>
 [00000060] Console or pseudoconsole

(Yes, <varargs.h> had to be replaced with <stdarg.h> for GCC)

This looks the same as in your example. In a WSL2 terminal and running it from Bash, I get

$ ./stdio.exe
<STD_INPUT_HANDLE>
 [0000005c] Console or pseudoconsole
<STD_OUTPUT_HANDLE>
 [00000060] Console or pseudoconsole
<STD_ERROR_HANDLE>
 [00000064] Console or pseudoconsole

I tried also some other things (change codepage to UTF-8, use simple example using fwrite etc.) but have no further idea.

Kind regards, Martin

(4) By Florian Balmer (florian.balmer) on 2023-09-19 16:04:28 in reply to 3 [link] [source]

Running the first three commands on CMD does not return anything.

Ok.

The output of the stdio.exe program in CMD is: ...

I'm sorry the fancy example stdio.exe < nul > %TEMP%\sample.log was to show the capabilities of the program. The important thing is that STD_OUTPUT_HANDLE is the same as STD_ERROR_HANDLE if stdio.exe is run without any redirects (the same as fossil.exe), as STD_ERROR_HANDLE seems to work. But I think this is also covered by the fossil.exe status 1>&2 check.

The <stdio.h> library wraps the standard IO handles provided by the OS, and so the next question is: is the file number/handle underlying stdout the same as the working one for stderr, and does it really point to STD_OUTPUT_HANDLE, or was is changed for some reason. With MSVC, this can be tested using:

HANDLE hStdout = (HANDLE)_get_osfhandle(_fileno(stdout));

(This test could be added to aSamples[] in stdio.c, with <stdio.h> included.)

If this is different from the handle returned by querying STD_OUTPUT_HANDLE directly, then the C runtime library for some reason has chosen to use another output handle.

Fossil mostly reuses what <stdio.h> provides, with the expection that UTF-8 output is converted to UTF-16 and sent directly to WriteConsoleW(), bypassing the <stdio.h> layer. A failed UTF-8 to UTF-16 conversion could maybe result in empty output, but this is again unlikely as the STD_ERROR_HANDLE path works.

I tried also ... simple example using fwrite ...

Does this mean something like this done in your x86_64-w64-mingw32 environment doesn't print any output, either:

#include <stdio.h>
main(){
  fwrite("Hello, world!",13,1,stdout);
}

This would indicate that the x86_64-w64-mingw32 C runtime library is not working correctly, and the problem is independent of Fossil.

(5) By smartmic on 2023-09-26 14:13:59 in reply to 4 [link] [source]

With MSVC, this can be tested using:

Unfortunately, I do not have MSVC installed and intend also not to do so (too much resources would be needed).

Does this mean something like this done in your x86_64-w64-mingw32 environment doesn't print any output, either:

Well, so along the lines of src/printf.c, I wrote:

#include<stdio.h>
#include<fcntl.h>
#include<io.h>
#include<windows.h>

int main() {
        char str[] = "hello world\n";
        _setmode(_fileno(stdout), _O_BINARY);
        fwrite(str, sizeof(char), sizeof(str) - 1, stdout);
        fflush(stdout);
        _setmode(_fileno(stdout), _O_TEXT);
        return 0;
}

But it prints output fine and I cannot (yet) blame x86_64-w64-mingw32. The next step would probably diving into the fossil_utf8_to_console function, but I probably find another way around my issue beforeā€¦

(6) By Florian Balmer (florian.balmer) on 2023-09-26 17:47:04 in reply to 5 [link] [source]

Unfortunately, I do not have MSVC installed and intend also not to do so (too much resources would be needed).

Yes, sure, I just meant you could find the x86_64-w64-mingw32 equivalent of MSVC's _get_osfhandle() to find out step by step where the <stdio.h> world deviates from the Win32 one.

The next step would probably diving into the fossil_utf8_to_console function ...

Maybe some _isatty() problem? Originally, the Cygwin terminal used pipes to emulate the standard handles (still available through ./mintty --pcon off), so _isatty() had its special heuristics to detect that case (IIRC).

But then again, this probably wouldn't explain the differences between stdout and stderr.