If they contain NULs then the routines which generate text/html output, e.g. the html diff view or diff console output, won't be able to render them properly.

Fossil's "this is binary" heuristics are baked into it, and are not configurable except to tell it not to warn about committing binary files which match user-defined globs.

i can't speculate how deep-seated the "NUL expectations" are - it's been a good 7 years since i dug around in those bits. The CLI and HTML diffs are definitely different implementations, as the former is line-based and the latter shows diffs with character-level precision.

The heuristics for "what is binary" also include files with exceptionally long lines, as fossil cannot generate readable diffs when the lines are thousands of bytes long.

(pardon my brevity - am on the phone in a waiting room.)

Can you describe the issue in a bit more detail? What type of source code is it? Is there some alternative representation that is supported?

For example, if the nul characters are embedded in strings that are C language compatible, they could easily be replaced by an equivalent octal, hexadecimal, or Unicode escape sequence which would not present a problem with fossil or arbitrary text editors or other tools.

That is often the type of issue when this question is asked. Using C-style escape sequences would make far more portable source files (if this is a C compatible string), even if the environment in question otherwise allows embedding a literal nul (or other traditionally non-printable character). Other languages might have comparable escape sequences or printable encoding in ASCII, UTF-8, or some other text encoding that would work with fossil.

Fair enough. Every time this has come up in the past that I remember reading it has been based on a C compatible string encoding.

What about representing #\<NUL> as (code-char 0) ?

Something like this:

; define constants for some non-printable ASCII chars
; (in case #\ equivalents are not available)
; and some sequences of such chars
(defconstant $/nul (code-char 0) "<NUL> like #\nul")
(defconstant $/us (code-char 31) "<US> like #\us")
(defconstant $/nul-us (list $/nul $/us) "<NUL><US> like #\nul #\us")

;example use of the defined constants
;instead of the explicit non-printable chars 
(print $/nul-us)

Tested with SBCL: https://ideone.com/9WInTP

Some Lisp implementations already have special non-printable ASCII chars defined beyond the standard ones like #\newlile, #\tab etc. I guess, this is for the exact same reason so that your source code remains printable and indeed compatible with the multitude of text-tools...It must be taking extra effort to even insert the non-printables into the text, let alone "seeing" them on-screen.

Well, anyways, I understand your frustration with this issue. Hope you'd find this helpful as a relatively straightforward work-around for the use-cases you've shown.

What is Fossil doing to these files with embedded 0x00 bytes that is causing problems for you?

It's what fossil's not doing: it won't diff binary files, and "contains NUL bytes" is (quite reasonably) one of Fossil's is-a-binary conditions.

By the same token, if the mentioned initial N bytes happen to include the "text" <NUL> that you use in your code, then we're back to square-one...

Clearly, the issue here is with a "hybrid" file that is not easy to reliably classify as binary/text in an automated way.

Fossil allows globbing for binary files to avoid the warnings and has an external diff as a workaround.
Seems fair to me.

By the way, the handling code is in src/lookslike.c looks_like_utf8() and invoked as either as such or via looks_like_binary() as in src/diffcmd.c

I have several files that require control characters, or long lines.

My solution is to "generate" such files, either from template files by replacing placeholders with the control characters, or directly by dumping parts of the long lines without line endings from a script. Then only the template files and/or scripts are checked into the repository, giving readable diffs.

Maybe something similar may work for you?