A bit tedious to analyse, but src/main.mk doesn't need that many changes.

Add explicit dependencies on $(OBJDIR) to these targets:

$(OBJDIR)/translate
$(OBJDIR)/makeheaders
$(OBJDIR)/mkindex
$(OBJDIR)/mkbuiltin
$(OBJDIR)/mkversion
$(OBJDIR)/codecheck1
$(OBJDIR)/shell.o
$(OBJDIR)/linenoise.o
$(OBJDIR)/th.o
$(OBJDIR)/th_lang.o
$(OBJDIR)/th_tcl.o
$(OBJDIR)/pikchr.o
$(OBJDIR)/cson_amalgamation.o

Remove no-longer-needed explicit dependencies on $(OBJDIR) from these targets:

all
test

Remove no-longer-needed -mkdir -p $(OBJDIR) command from the $(OBJDIR)/translate target.

All the translated source files already depend on $(OBJDIR)/translate, so no change needed there.

Add explicit dependencies on $(OBJDIR) to these targets:

Is that right? Because then whenever a new file is added to $(OBJDIR), the mtime of that directory will change and require all of those targets to be rebuilt, no?

cc -I. -I./src -I./extsrc -Ibld -Wall -Wdeclaration-after-statement -DFOSSIL_ENABLE_JSON -DFOSSIL_ENABLE_TH1_DOCS

Unrelated: is there a particular reason that build is done with --json and --enable-th1-docs? Neither of those are enabled in the canonical releases and the latter is considered an "iffy" feature which we'd prefer that people willfully activate if they insist on using it (all the while recommending that they don't).

I may have posted while insufficiently awake.

Just failure on my part to keep up with the times.

Here's what I have now:

override_dh_auto_configure:
        dh_auto_configure -- --disable-option-checking \
                             $(FOSSIL_CONF_SQLITE_OPTIONS) \
                             --json \
                             --with-th1-docs --with-th1-hooks \
                             --with-tcl=1 \
                             --with-tcl-stubs \
                             --with-tcl-private-stubs

What should it be?

For an OS package, IMO (but only IMO) it should match the canonical releases unless there's a compelling reason not to, which would be simply:

  --disable-fusefs

The canonical builds are built with an explicit OpenSSL version, but that's not necessary: configure will fail if it cannot find SSL unless --with-openssl=none is used (which you don't want to do).

I'm actually building with fusefs, but getting some push-back on it because fossil uses an old version (2) of the fuse library that they're trying to get things to migrate off of.

And there's a --disable-internal-sqlite hidden in there too.

These are the current build dependencies:

               libsqlite3-dev (>= 3.49),
               bash-completion,
               libfuse-dev,
               libreadline-dev,
               pkgconf,
               libssl-dev,
               tcl | tclsh,
               tcl-dev,
               zlib1g-dev | libz-dev

The --with-th1-docs option is a built-in security vulnerability. It allows users with check-in privilege on a repository to break into the system that hosts the remote repository. I recognized this as soon as the originator of that feature started lobbying for it to land on trunk, but I was told "It's a compile-time option that defaults OFF and I need it for my project." I relented and let it go in. I now recognize that was a mistake. I apologize. (The most important, and also the most difficult, job of a project coordinator is saying "no!".)

Please do not publish versions of Fossil that are compiled with --with-th1-docs.

Note: In addition to the compile-time option, there is also a per-repository configuration option that must be turned on in order to enable the th1-docs feature. There is a two-step process to activate the security vulnerability, and the compile-time option is only the first of those two steps. So it is not like all previously published versions of Fossil are vulnerable. They just have one fewer layer of defense than if they were built using the default configuration.

I am less clear on all of the other options you have enabled, in as much as I don't use them nor do I know of people who are using them. I am guessing that they are less secure than the default configuration. To maximize safety and to minimize the chance of zero-day exploits, I recommend building the default configuration. Omit TCL. Omit TH1 docs and hooks. Omit Json extensions. Include OpenSSL, obviously, but leave everything else off. The few users who depend on those other (unsupported) enhancements should be capable of building Fossil for themselves.

Will do!

Also, as a point of clarification.

It is the general Debian policy to compile software with as many configuration options enabled as practical, to avoid users needing to recompile things. I suspect that's a policy shared with many other packaging ecosystems. So if that's not appropriate, it might be a good idea to put that information somewhere prominent.

... to avoid users needing to recompile things. ... it might be a good idea to put that information somewhere prominent.

FWIW, i'd argue that the instructions for the canonical builds (which are, by definition, for those folks who don't want to build it themselves) implicitly provide that info.

Maybe we should remove the options from ./configure and require users who want to make use of those rogue options to explicitly add the appropriate -D arguments on the command-line? I would be OK with that.

Maybe just distinguish the not-recommended options in ./configure --help as experimental or unstable or currently under development or something. And if --disable-fuse is the recommended setting, it should be the default, yes?

And what about Emscripten; should I enable that?

I know everyone hates on autoconf, but it does automatically label the options as to their default values in its --help text.

And what about Emscripten; should I enable that?

Emscripten is only needed when updating fossil's embedded copy of pikchr.c from its upstream source, in order to rebuild its wasm file (which is checked in, so only people who do such updating need Emscripten).

I know everyone hates on autoconf, but it does automatically label the options as to their default values in its --help text.

For better or worse, there's a difference between "the defaults" and "what gets used for canonical builds." Perhaps there shouldn't be.

The help system will show values for flags which have them, but it does not for booleans. Boolean flags are off by default, and any given boolean flags which are explicitly set to default to true will show up as --disable-that-flag in the help text. (That all happens behind the scenes from us - we have no influence on it beyond to patch autosetup.)

For better or worse, there's a difference between "the defaults" and "what gets used for canonical builds." Perhaps there shouldn't be.

While it may make sense to have the same, the goal of a ./configure script is to detect what is available on your system and use it. While for the Canonical build, which end up in the fossil download page is specifically built with less dependency possible (e.g. --disable-fuse) with static openssl so the binary can run on the majority of the system.

I would strongly support that too, and switching to --enable-fusefs. And possibly a larger-scale overhaul,why we are at it.

Emscripten is only needed when updating fossil's embedded copy of pikchr.c from its upstream source, in order to rebuild its wasm file (which is checked in, so only people who do such updating need Emscripten).

Urg. Debian Policy is that /everything/ should be rebuilt from source, so that would include rebuilding the wasm file from the embedded (or in the strange nomenclature kids use these days, the ‘vendored’) pikchr.c. I'll put it on my to-do list.

Regarding configuration options, how about a middle road? Options that are dangerous or unstable or not recommended could require a ./configure --force option, or --experimental, or something like that. Trying to enable them without that option would result in a configuration-time error. This would maintain the property of there being one place for configuration options, but would be a bit of a guard rail to protect idiots like me from myself.

Urg. Debian Policy is that /everything/ should be rebuilt from source, so that would include rebuilding the wasm file from the embedded (or in the strange nomenclature kids use these days, the ‘vendored’) pikchr.c. I'll put it on my to-do list.

Don't do that. If you do that, you'll need to ensure that your copy of pikchr.wasm is built identically to the one in the canonical tree (which we check in)¹. Also, Emscripten makes breaking changes from time to time, so you have to ensure that the Emscripten you use is behaviorally compatible with the one which was used by the checked-in copy of pikchr.

extsrc/pikchr.wasm and extsrc/pikchr.js are checked in to the source tree for a reason: namely, so that nobody except for the two people who update extsrc/pikchr.c from its upstream copy have to deal with Emscripten. When pikchr.c is updated, pikchr.js/wasm are always rebuilt and checked in at the same time (because if they're not, /pikchrshow embeds a different pikchr version than the fossil binary does). Then, if an Emscripten change broke something, the app-level code which uses pikchr.wasm is adapted to suit. That last step cannot be accounted for if you generate the wasm/js on your own.

There's no reason, beyond a love of ice-skating uphill, for anyone to have to independently generate pikchr.wasm/js.

1. ^{^} Covered in what is currently section 8.0 of https://fossil-scm.org/home/doc/trunk/www/build.wiki.

Okay.

(a) I won't try to regenerate the pikchr wasm.

But (b)

There's no reason, beyond a love of ice-skating uphill, for anyone to have to independently generate pikchr.wasm/js.

that said, these black magic “nobody else can regenerate this correctly without secret insider knowledge we will take to our graves” vibes are pretty much /why/ Debian has adopted a "regenerate it /all/" policy. And pikchr.y is nowhere to be found; that's the true source, from which pikchr.c and pikchr.js are ultimately built, right? (Well, it's on github so not exactly obscure; but not found /here/.) I guess I'm saying “it's tricky and we have not automated the build process or even fully documented the manual steps we take” is not exactly reassuring to people who worry about vulnerabilities hidden in pre-compiled code. I note that pikchr build documentation is available on the pikchr site, and seems pretty complete. But there's some tricky special stuff going on with your particular build for fossil, if I'm understanding you correctly?

We want the average programmer to be able to easily compile Fossil for themselves. That means download, then run "./configure && make". And for that case, it makes sense to include a prebuilt pikchr.wasm in the source tree, as building pikchr.wasm from source requires a lot of specialized tooling that most developers do not have readily at hand.

But distributions like Debian have a different set of requirements. I completely understand that they want to compile everything from original sources, if for no other reason than just the security of being able to inspect and/or edit everything.

The information on how to build everything from sources is available in the documentation now, I think. But it is perhaps not as well organized as it should be. Perhaps we should put together a new documentation page about "How To Build Everything From Sources" that is focused specifically on the best way to do exactly that?

But there's some tricky special stuff going on with your particular build for fossil, if I'm understanding you correctly?

Tricky, no, until Emscripten changes something. Several times, after updating, we've had to go update /pikchrshow code because an Emscripten update changed its behavior at a level which broke that app. The only way to know if /pikchrshow still works after re-building the wasm file is to build fossil, start up pikchrshow, and manually check that it works. That's in appropriate for both downstream users and automated build/packaging processes.

Including pikchr.wasm/js in the tree is not just so that users don't have to have Emscripten, but also so that they don't have to have a version compatible with whatever the /pikchrshow app-level code was written to work with. We can't sensibly support arbitrary Emscripten versions in that code, so can't reasonably support people building with whatever Emscripten they want to use.

There's no reason, beyond a love of ice-skating uphill, for anyone to have to independently generate pikchr.wasm/js.

Unless somebody is using source inspection / diffs to protect themselves from malware injection attacks?¹ If pikchr.wasm/js are human readable (in the sense of being understandable, rather than just being in text) then you are fine. If not, then the debian package builder needs to look at two packages rather than one, or replicate the pikchr source and build within their archive of Fossil source. They will also need to establish trust in the Emscripten tool chain.

I like the ice-skating simile. I suspect cyber security can indeed feel like that.

1. ^{^} I am not offering any opinion about how effective a source diff strategy is.

Debian Policy is that /everything/ should be rebuilt from source

Thinking more about this... The pikchr.wasm file is not actually something used by Fossil itself. Pikchr.wasm is a resource file that Fossil sends down to the web browser when the web browser requests it. So in some sense, pikchr.wasm is more like a JPEG or PNG file than code that needs to be compiled.

True, pikchr.wasm is executable on the browser, but JPEG and PNG files are too. Granted, JPEG and PNG are not Turing-complete like WASM files, so there is some difference. Nevertheless, they are all just resource files that are handed down to the web browser. Pikchr.wasm lives in the gray region in-between pure executables that need to be built from sources, and resource files like images that are just data.

Should the pikchr.wasm file be run-time configurable?

Various skins are built into Fossil and most people use one of those built-in skins, but an administrator can generate their own custom skin without having to rebuild Fossil. Should pikchr.wasm be the same? Should someone with admin privilege be allowed to change a setting that specifies an alternative pikchr.wasm file?

The pikchr.wasm file is not actually something used by Fossil itself. Pikchr.wasm is a resource file that Fossil sends down to the web browser when the web browser requests it.

Indirectly, fossil does use it and relies on its precise build conditions, via the /pikchrshow app-level code.

Should pikchr.wasm be the same? Should someone with admin privilege be allowed to change a setting that specifies an alternative pikchr.wasm file?

Nope - they risk breaking /pikchshow if they do.

Should the pikchr.wasm file be run-time configurable?

Not meant to be heretical, but pikchr.wasm was created as a proof-of-concept and exploring how to create the SQLite fiddle, no? Can't we just drop pikchr.wasm and make /pikchrshow?legacy the default, again? Or at least add a build option to exclude pikchr.wasm and only use /pikchrshow?legacy? Like this, non-Wasm-trained people could still build everything from source, themselves, without losing any functionality.

(Disclaimer: I have disabled Wasm for all my web browsers for security concerns.)

I tend to agree here. It seems unnecessary to include by default a fatty wasm and bring with it a flock of problems, when there's a perfectly working alternative. I like pikchr and having built-in support for it in Fossil is great, yet doing it with wasm rather overshoots the mark.

Or am I missing something and the wasm version adds significantly many and/or important needed features that the legacy one does not?

Or am I missing something and the wasm version adds significantly many and/or important needed features that the legacy one does not?

Most significantly, as-you-type updating of the pikchr. Secondly, no server load after the page loads.

We couldn't do the "click here to open in pikchrshow" links with server-side pikchr unless we changed those from plain hyperlinks to POST requests, and then i'm not certain we could get them to open in a new window:

box

Most significantly, as-you-type updating of the pikchr.

The "Render while typing" feature may be useful languages/scripts mostly containing plain text and only little markup, such as Markdown.

But for pikchr, consider the following statement (from one of the Example Scripts):

C: circle "center" rad 150%

While typing this line from left to right without any typos, there's 6 valid states:

C: circle "center" rad 150%
C: circle "center" rad 150
C: circle "center" rad 15
C: circle "center" rad 1
C: circle "center"
C: circle

And there's 21 invalid states resulting in the yellow "synatx error" page.

So I don't think "Render while typing" is really useful, here.

... POST requests, and then i'm not certain we could get them to open in a new window ...

Maybe like this:

<form target="_blank">

It doesn't really matter to me whether the pikchr.wasm stays or not. I see the heart and soul invested and the technical finesse, but also the "overkill factor" and the blocking obstacles for non-experts to rebuild it, themselves, with reasonable efforts.