Fossil Forum

Forum
Login

Automatic Code Highlighting with Prism

(1) By wyoung on 2019-09-02 20:38:15 [link]

One of my public Fossil repos now does automatic code syntax highlighting using Prism, a particularly nice and modern example of the art. In this posting, I will show how I did it, and explain how you can do it with your repos.

Demos

How-To

Make these two simple additions to your Fossil skin:

  1. Add this one-liner to the Footer section:

    <script nonce="$<nonce>" src="/js/prism.js"></script>
    
  2. In the CSS section, append the contents of the block above the "DOWNLOAD CSS" button at the bottom of the download page. That CSS is dynamically generated from your selections above it, so you might want to adjust the settings in that link, which are suitable for my repo, but maybe not yours.

    I used the "Coy" Prism theme for the demos above, since it works well with the default Fossil skin, which I'm using with few adjustments on that site.

Now you need to arrange for Prism's Javascript to be served within the restrictions of the default CSP. There are a few ways:

  1. The way I did it is dependent on the fact that my public Fossils are served as virtual subdirectories of my otherwise static web site. This allowed me to create a /js subdirectory off the root of the static part of my public web site, and put the prism.js file there.

  2. If you don't have a front-end HTTP server that allows you to serve static files like this, you could store prism.js in your repo as unversioned content and point to it thus:

    <script nonce="$<nonce>" src="$<root>/uv/prism.js"></script>
    

Served File Size Bloat?

You might not be happy with the CSS edit above, which roughly doubles the default size of the Fossil-provided virtual style.css file.

This is a non-issue if you have a sufficiently smart front-end proxy in front of Fossil. With it, you can match the style.css virtual file by name and apply a far-future expires header on it, allowing that file to be cached indefinitely once pulled. Since Fossil gives a new version to the file each time it changes, there's no worry over stale CSS even with a multi-year cache expire time.

In this way, you can cause browsers to pull it once on first visit and then never need to pull it again until it changes, or they stop visiting often enough that the file stays in their cache.

You want to be doing this anyway, Prism or no. Properly tuned, a web site will pull such CSS files virtually instantaneously from cache, rather than over the World Wide Wait every time.

The code-in-pre Feature

If you do the above steps to your site, only Markdown fenced code blocks will be styled like this.

The second demo above relies on a new feature I've just added to Fossil on the code-in-pre branch. This feature causes Fossil to put <code class="language-EXT"> inside the <pre> tags it uses for /artifact and such, with EXT being the extension of the file content being shown. We rely on Prism to guess the correct syntax highlighting rules from that extension.

Keep in mind that I'm writing this only minutes after checking in the first functional version of that feature. I have no reason to expect that it covers all cases. I'm also not wild about how it does a file name lookup in the Fossil SQL DB purely for this purpose; I was hoping Fossil had already gathered that info somehow above the HTML code I added to the Fossil source, but I couldn't see anywhere I could snag it, so I gave up and added my one-off SQL.

Future Directions

I'm toying with the idea of making this posting the seed of a new section in the docs, "Integrating Fossil with..." using a directory structure like www/int/syntax/prism.md, with the idea that the section will grow to encompass dozens or even hundreds of documents as people come up with local integrations. Between skin tweaks, /ext, and TH1, we should be able to fill this section quickly, contributors willing.

(2) By ckennedy on 2019-09-02 21:07:08 [link] in reply to 1

This is seriously cool Warren. I tried to get this working once with Fossil using some info I found online somewhere and couldn't get it to work at all. The above is quite simple and doable.

I would suggest the new section of the docs be called "Extending Fossil with..." rather than "Integrating Fossil with...". To my mind this fits better with the CGI extension feature as well as what is actually happening. We are extending Fossil with other tools/features.

Thanks.

(3) By anonymous on 2019-09-02 21:20:25 [link] in reply to 1

Very cool. Thanks for the work you have already done on the docs. This looks like a start to a new series of documents 8-).

IIRC the cgi root directory will just serve up files if they don't have the execute bit set. So you should also be able to:

  1. store prism.js in your CGI server extension directory. E.G. /home/fossil/cgi
  2. start fossil with the argument: -extroot: /home/fossil/cgi

and point to it with:

 <script nonce="$<nonce>" src="$<root>/ext/prism.js"></script>

The only question I have is whether the mime type served up by accessing the file this way would be correct and allow the code to be executed.

Sadly my server upgrade went sideways today, so my fossil repos are offline and I can't test.

(4) By wyoung on 2019-09-02 23:34:37 [link] in reply to 3

start fossil with the argument: -extroot: /home/fossil/cgi

That prompted me to write "Serving Files Within the Limits," an extension to the default CSP doc, which adds that idea and expands on what we already covered. The overall message I hope readers get from that section is, "You probably don't have to override the default CSP."

the mime type served up

Does Fossil not reuse its internal MIME type guesser from the embedded docs feature for that?

I was surprised by the very notion that the MIME type even mattered when it came to running JS in modern browsers. Once upon a time, sure, but JS has wiped out all of the competition so that the only alternatives that survive are transpiled to JS to make them run.

According to one source, you don't need a MIME type as long as the HTML includes the now-nonstandard type="javascript" attribute. But, Fossil doesn't include that attribute, so for us, MIME type does apparently matter.

(5) By wyoung on 2019-09-03 00:45:02 [link] in reply to 1

I've made some refinements to the JS for the above technique:

<script nonce="$nonce" src="$<home>/file/src/misc/prism.js?download"></script>
<script nonce="$nonce">
  Prism.languages.def = Prism.languages.tcl;
  Prism.languages.fc  = Prism.languages.bas = Prism.languages.basic;
  Prism.languages.ft  = Prism.languages.fortran;

  (function() {
    function iscrlf(c) { return c == 10 || c == 13 }
    document.querySelectorAll("pre > code").forEach((e) => {
      var h = e.innerHTML;
      while (h.length > 2 && iscrlf(h.charCodeAt(h.length - 1))) {
        h = e.innerHTML = h.substring(0, h.length - 2);
      }
    })
  })();

  <th1>styleScript</th1>
</script>

  1. It shows use of an in-repo /file URL instead of an out-of-repo /js URL. Note the use of the TH1 variable $home, since the repo I did the work on is not serving the whole site, only one "subdirectory."

    (By the way, I tried a /raw URL first, as it's shorter, but that caused the JS file to be downloaded to disk. This might be the MIME type issue brought up elsewhere in this thread.)

  2. It shows document type aliasing for cases where Prism can't detect the proper type from the file name extension alone. Yeah, we've got FORTRAN IV highlighting now, baby!

  3. Automatic removal of any CR or LF at the end of the file, which forces the closing </code> tag to the next line in the HTML, which means the <pre> wrapping it creates a faux newline at the end of the highlit code block. The inline JS above fixes that.

    Note that I've just inserted this code block into the one that normally calls styleScript from the Footer in the default skin.

(6) By achavasse on 2019-09-03 08:28:40 [link] in reply to 1

The "code in pre" feature is nice. I use a secondary script as a way to map file types to languages to highlight which is much clumsier, although it's an ok workaround until that branch is merged.

I'm wondering if for the sake of accessibility it might be nice to have premade scripts that you could run on a fossil repo to automatically setup syntax highlighting, or other similar js based features, like the graph generator mentioned elsewhere, so that a new user could easily add those features to their repo if they wanted.

(7) By anonymous on 2019-09-04 03:10:52 [link] in reply to 1

Looks nice!

I tried to see if it Artifact's hilighting also works with line numbers on... Does not seem to kick in. Is it just my browser or that's how it is?

(8) By wyetr on 2019-09-04 05:09:01 in reply to 7

Yup, that breaks it. The inner <code> tags aren't getting inserted in that case, which Prism requires. I'll look into fixing it, maybe tomorrow, unless someone beats me to it.

(9) By wyoung on 2019-09-05 01:38:37 [link] in reply to 8

I've fixed the immediate symptom on the new code-in-pre-with-ln branch, but it's got a number of problems:

  1. If you just want line numbers, use the Prism line numbers plugin.

  2. If you're turning on line numbers to use Fossil's line highlighting feature — e.g. ?ln=42-69 — applying this branch's change causes Prism to override it, effectively breaking the Fossil feature.

  3. The line numbers emitted by Fossil are intermixed with the code, so that a syntax highlighter is likely to see them as syntax errors.

To fix all of this, you'd want a much more intelligent integration. The Fossil and Prism line numbering features would have to cooperate instead of fight, the line numbers would have to be outside the <code> block holding the actual code so they don't interfere, and there would have to be a way for Fossil's line highlighting code to integrate with Prism's syntax highlighting.

If someone wants to take that project on, you're welcome to it. It won't be me.

(10) By anonymous on 2019-09-05 06:01:43 [link] in reply to 9

Can Prism deal with statements being split across <code> spans?

If so, maybe this would work:

<pre>
     1  <code>line of code</code>
     2  <code>another line of code</code>
     3  <code>and more code</code>
</pre>

I know that's a lot of extra markup.

(11) By wyoung on 2019-09-05 08:21:40 [link] in reply to 10

I assume that would cause it to lose context and thus create a bunch of false syntax errors. For example, how could it be expected to deal with this bit from Fossil's own source code:

    zCmd = mprintf("\"%s\" http --in \"%s\" --out \"%s\" --ipaddr 127.0.0.1"
                   " \"%s\" --localauth",
       g.nameOfExe, transport.zOutFile, transport.zInFile, pUrlData->name
    );

I'd suggest one of two other solutions:

<table>
  <tr>
    <td>
      <pre>
        1
        2
        3
      </pre>
    </td>
    <td>
      <pre><code>first line of code
      second line of code
      third line of code
    </td>
  </tr>
</table>

If table-based layouts are deemed too horrible, then there's a straightforward method involving CSS and <div> elements with the same basic content.

Either way, you'd have to style it so that the lines of code never wrap, even if it means horizontal scrolling.