For writing documentation using Markdown, I'm finding that I want to have the option for the following automatically generated features:

There is a considerably-lengthy discussion about this, and some of the hurdles, over at /forumpost/097eca7dd9.

Related: someone recently posted a link to a free pure-JS impl which works on arbitrary HTML at /forumpost/6aa9e9e82. i've poked at that one a bit and squirreled it away for my own use later on - it seems to be pretty decent as a starting point, in any case.

For writing documentation using Markdown, I'm finding that I want to have the option for the following automatically generated features

It's a pain and no I don't believe there is a markdown standard. I rate Pandoc transformations highly, and Pandoc has developed Markdown extensions including tables described as:

Pandoc understands a number of useful markdown syntax extensions, including document metadata (title, author, date); footnotes; tables; definition lists; superscript and subscript; strikeout; enhanced ordered lists (start number and numbering style are significant); running example lists; delimited code blocks with syntax highlighting; smart quotes, dashes, and ellipses; markdown inside HTML blocks; and inline LaTeX. If strict markdown compatibility is desired, all of these extensions can be turned off. LaTeX math (and even macros) can be used in markdown documents.

So those Pandoc Markdown extensions would be something to investigate, I have not looked into it. I will though, because I like the idea of solving this in a better way.

To date for standalone docs I have avoided the Markdown metadata problem for TOCs in two ways:

1. Use Pandoc to translate from Markdown to Markdown with --toc

2. Use the crude-but-works script that will generate markdown that can be inserted with a script by calling public github APIs.

Dan

So those Pandoc Markdown extensions would be something to investigate, I have not looked into it.

Here is the Pandoc Markdown manual .

Dan

I rate Pandoc transformations highly

FWIW, that was a general consensus buried somewhere in the older post linked to in my first response. Pandoc seems to have done a commendable job on that.

A couple of thoughts:

1. You need a way to cut off ToC generation at an arbitrary header level. h1 and h2 are fine for this doc, h1-h3 make more sense for that doc, h1-h5 almost never makes sense, etc.

2. Where do you put the ToC once you have it?

You need a way to cut off ToC generation at an arbitrary header level.

Where do you put the ToC once you have it?

GoogleCode wiki format (my favorite until markdown finally won me over) answered both of those by requiring the doc author to add a tag to the wiki doc:

<wiki:toc max_depth="number"></wiki:toc>

Something similar sounds like a reasonable idea for us. That said: in forum posts we probably don't want them output at all, if for no other reason than tocs in multiple posts in the same thread may well collide in terms of reference IDs.

I broke the parser with my YAML example :-)

Dan

I would go with YAML metadata blocks, as proposed by Dan and practiced by Pandoc and a lot more of Markdown based wiki/blog/CMS engines.

I don't know if adding a YAML parser to Fossil Markdown parser is a lot of overhead for this feature, but certainly it will easy adoption and extension of Markdown features, as is a practice so widely used. Between users of such extension, a key/value pair is expected to control such custom behavior (something like toc: true).

My second option would be something with Mustache alike based syntax ({{toc}}) as proposed by Aslak, following MultiMarkdown.

Both syntax are easy and open the door for further customization options (like TOC location or levels to show), but in that case I think that YAML is easier to read and write, as nested configurations are usual there.

Cheers,

I don't know if adding a YAML parser to Fossil Markdown parser is a lot of overhead for this feature

Looking at the proposal, seems like YAML Tiny would be sufficient to support the meta data block. Maybe this subset has low enough overhead to be worth implementing.

The specification is at https://metacpan.org/pod/YAML::Tiny#YAML-TINY-SPECIFICATION, in the documentation of its reference implementation.

... automatically generated features.

This breaks down into a number of subproblems, some of which have already been solved. For example, re-using an existing automatic heading id algorithm is probably sensible.

On the other hand, both PHP Markdown Extra and Pandoc Markdown also allow manual heading ids by applying classes like this:

{#identifier .class .class key=value key=value}

An extensive numbering explanation goes into great detail.

paragraph numbering

I think this is standard Markdown, if you mean to have paragraphs as numbered lists. So long as you keep indenting by at least one space after a numbered sentence, then it is the same paragraph. So long as there is no more than one line between paragraphs, then numbering does not reset. And voila, numbered paragraphs.

paragraph ids

I do not think this is standard but doesn't seem a big leap to implement this given that both Pandoc Markdown and PHP Markdown Extra allow classes to be applied to objects like tables and code blocks. Eg here is a PHP Markdown Extra code block with class applied from the docs:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.html #example-1}

paragraph emphasis ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

So how about just applying this to a paragraph?

A tidy solution to metadata in Markdown that Pandoc implements is a YAML block that can appear anywhere in the document. I won't try to quote this since I just had a battle with the forum parser and lost my post ;-) I don't know if that is where TH1 should replace YAML, I don't know anything much about TH1.

Dan

Here's what I've been doing for years. Maybe you might find some inspiration.

#!/usr/bin/env tclsh

# Load required packages.
package require Tcl 8.6

# encHex --
# Encode by substituting most non-alphanumerics with hexadecimal codes.
proc encHex {str {pattern {[^-^,./'=+|!$\w]}}} {
    set pos 0
    while {[regexp -indices -start $pos $pattern $str range]} {
        binary scan [string range $str {*}$range] H2 char
        set str [string replace $str {*}$range %$char]
        set pos [expr {[lindex $range 0] + 3}]
    }
    return $str
}

# parseMarkdown --
# Basic Markdown parser that supports only a small subset of Markdown.
proc parseMarkdown {str} {
    # Initialize intermediate and result variables, then invoke parser.
    set index 0
    set html {}
    set plain {}
    apply {{{emph {}}} {
        # Get access to caller variables.
        upvar 1 str str index index html html plain plain

        # Helper procedure that appends a literal string to the result.
        set literal {apply {{{count 1}} {
            if {$count > 0} {
                upvar 1 str str html html plain plain index index
                set start $index
                incr index $count
                if {$index > [string length $str]} {
                    set index [string length $str]
                }
                set chunk [string range $str $start [expr {$index - 1}]]
                append html [string map {
                    < &lt; > &gt; & &amp; _ \\_ * \\* ` \\` [ \\[ ] \\]
                } $chunk]
                append plain $chunk
            }
        }}}

        # Find the next supported Markdown introductory metacharacter.
        while {[regexp -start $index -indices {[<&`*_\\]} $str match]} {
            # Emit literal text since the start or the prior sequence.
            {*}$literal [expr {[lindex $match 0] - $index}]

            # Process this special sequence according to its first character.
            switch [string index $str $index] {
            < - & {
                # Directly emit literal HTML tags and entities.
                if {[regexp -start $index -expanded {
                    \A<(?:[^'\">]+|'[^']*'|\"[^\"]*\")+>    # HTML tag
                   |\A&[^;]+;                               # HTML entity
                } $str match]} {
                    # Parse success.  Emit the HTML.
                    append html $match
                    incr index [string length $match]
                } else {
                    # Parse error.  Emit the character literally.
                    {*}$literal
                }
            } ` {
                # Emit backtick-quoted text.
                append html <code>
                if {[regexp -start $index {\A`.*?`} $str match]} {
                    # Parse success.  Emit the backtick-quoted text literally.
                    incr index
                    {*}$literal [expr {[string length $match] - 2}]
                    incr index
                } else {
                    # Parse failure.  Emit the backtick literally.
                    {*}$literal
                }
                append html </code>
            } * - _ {
                # Emit emphasized text.
                if {$emph eq {}} {
                    # Beginning of emphasis.  Recurse into the parser.
                    regexp -start $index {\A(.)\1{0,2}} $str nest
                    incr index [string length $nest]
                    switch [string length $nest] {
                        1 {append html <em>}
                        2 {append html <strong>}
                        3 {append html <strong><em>}
                    }
                    {*}[info level 0] $nest
                } elseif {$emph eq [string range $str $index\
                        [expr {$index + [string length $emph] - 1}]]} {
                    # End of emphasis.  Stop the recursive parser.
                    switch [string length $emph] {
                        1 {append html </em>}
                        2 {append html </strong>}
                        3 {append html </em></strong>}
                    }
                    incr index [string length $emph]
                    return
                } else {
                    # Parse error.  Emit the character literally.
                    {*}$literal
                }
            } \\ {
                # Emit the backslashed character literally.
                incr index
                {*}$literal
            }}
        }

        # Emit literal text following the final sequence.
        {*}$literal [expr {[string length $str] - $index}]
    }}

    # Return the HTML and plain text.
    list $html $plain
}

# Check command-line arguments.
if {![llength $argv]} {
    set notoc 0
} elseif {[llength $argv] > 1 || [lindex $argv 0] ne "-notoc"} {
    chan puts stderr "Usage: [file tail $argv0] ?-notoc?"
} else {
    set notoc 1
}

# Find documentation directory.
set dir [file join [file dirname [info script]] doc]

# Read titles and other information from all input files except index.md.
set fileTitle {}
set orderGroup {}
set groupFiles {}
foreach file [glob -directory $dir *.md] {
    if {[set tail [file tail $file]] ne "index.md"} {
        # Read file.
        set chan [open $file]
        set data [chan read $chan]
        chan close $chan

        # Extract information from file.
        dict set fileTitle $tail [lindex [regexp -inline -line {.*} $data] 0]
        if {![regexp -line {^<!--GROUP=(\d+),(.*)-->$} $data _ order group]} {
            error "group definition not found in $tail"
        } elseif {![regexp -line {^<!--ORDER=(\d+)-->$} $data _ fileOrder]} {
            error "file order not found in $tail"
        }

        # Save group ordering.
        if {![dict exists $orderGroup $order]} {
            dict set orderGroup $order $group
        } elseif {[dict get $orderGroup $order] ne $group} {
            error "inconsistent group names \"$group\" and\
                    \"[dict get $orderGroup $order]\" for group order $order"
        }

        # Add file to group.
        if {![dict exists $groupFiles $group $fileOrder]} {
            dict set groupFiles $group $fileOrder $tail
        } else {
            error "file order $fileOrder in group \"$group\" used for both\
                    \"$tail\" and \"[dict get $groupFiles $group $fileOrder]\""
        }
    }
}

# Create documentation index file and header.
set chan [open [file join $dir index.md] wb]
chan puts $chan\
{Documentation Index
===================
<!--This file is generated by docindex.tcl-->
<!--TOC-->

# Search Documentation

<form action="$ROOT/docsrch" method="GET">
  <input type="text" name="s" size="40" autofocus="1">
  <input type="submit" value="Search">
</form>}

# Process each group in order.
foreach {_ group} [lsort -integer -stride 2 -index 0 $orderGroup] {
    # Emit the group heading.
    chan puts $chan "\n# $group\n"

    # Emit the document list in order.
    foreach {_ tail} [lsort -integer -stride 2 -index 0\
            [dict get $groupFiles $group]] {
        chan puts $chan "- \[[dict get $fileTitle $tail]\]($tail)"
    }
}

# Add version selection and Vim modeline, then close the documentation index.
puts $chan {
# Alternate Documentation Versions

- [Most recent check-in on trunk](/doc/trunk/doc/)
- [Most recent check-in on any branch](/doc/tip/doc/)
- [Current checkout](/doc/ckout/doc/)

<!-- vim: set sts=4 sw=4 tw=80 et ft=markdown: -->}
chan close $chan

# Link back to table of contents.
set top\
{<a href="#table_of_contents" style="font-size:small;float:right">[top]</a>}

# Process each documentation file.
foreach file [glob -directory $dir *.md] {
    # Read Markdown file.
    set chan [open $file]
    set data [chan read $chan]
    chan close $chan

    # Skip the file if it does not contain a TOC.
    if {[string first <!--TOC--> $data] < 0} {
        continue
    }

    # Identify all code blocks fenced by "```" lines.  Permit the opening code
    # fence to be followed by other characters, e.g. syntax mode name.
    set fences [regexp -indices -line -inline -all {^```(?:.|\n)*?^```$} $data]

    # Initialize dict listing all existing anchors, used to avoid duplication.
    set anchors {table_of_contents {}}

    # Build the TOC to contain all first- and second-level headings.  Consider
    # only headings using "#" and "##" marks, not underlines, and skip headings
    # that appear inside fenced code blocks.
    set start 0
    set offset 0
    set toc <!--TOC-->
    set oldHeading #
    if {!$notoc} {
        append toc "\n<span id=\"table_of_contents\"></span><div class=\"toc\"><ul>"
    }
    while {[regexp -indices -line -start $start {^##?[^#].*} $data match]} {
        # Place the start and end indices in their own variables.
        lassign $match match0 match1

        # Skip this match if it starts within a fenced code block.
        set skip 0
        foreach fence $fences {
            if {$match0 >= [lindex $fence 0] + $offset
             && $match0 <= [lindex $fence 1] + $offset} {
                set skip 1
                break
            }
        }
        if {$skip} {
            set start [expr {$match1 + 1}]
            continue
        }

        # Get line from input.
        set line [string range $data $match0 $match1]

        # Get heading level.
        regexp {^(#*)(.*)} $line _ heading line

        # Strip anchor tag, TOC link, and enclosing whitespace.
        regsub {^ <span id=".*"></span>} $line {} line
        regsub { <a href="#.*".*} $line {} line
        set line [string trim $line]

        if {$notoc} {
            # Strip anchor and TOC link from the section header line.
            set line "$heading $line"
        } else {
            # Extract the title and convert to HTML and plain text.
            lassign [parseMarkdown [string trim $line]] html anchor

            # Compute unique anchor name.
            set anchor [regsub -all {\W+} [string tolower $anchor] _]
            set anchor [encHex [string trim $anchor _]]
            if {[dict exists $anchors $anchor]} {
                for {set i 1} {[dict exists $anchors $anchor\_$i]} {incr i} {}
                append anchor _$i
            }
            dict set anchors $anchor {}

            # Build table of contents.
            if {$heading eq $oldHeading} {
                append toc \n
            } elseif {$heading eq "#"} {
                append toc </ul>\n
            } else {
                append toc \n<ul>
            }
            set oldHeading $heading
            append toc "<li><a href=\"#" $anchor \"> $html </a></li>

            # Add an anchor and a TOC link to the section header line.
            set line "$heading <span id=\"$anchor\"></span> $line $top"
        }

        # Replace the section header line.
        set data [string replace $data $match0 $match1 $line]

        # Update the start index and offset.
        set start [expr {[string length $line] + $match0}]
        incr offset [expr {$start - $match1 - 1}]
    }
    if {!$notoc} {
        append toc \n</ul></div>
    }

    # Write Markdown file with the table of contents inserted or removed.
    set chan [open $file wb]
    chan puts -nonewline $chan [regsub -line {^<!--TOC-->$(?:\n.+$)*\n$}\
            $data [string map {& \\& \\ \\\\} $toc]\n]
    chan close $chan
}

# vim: set sts=4 sw=4 tw=80 et ft=tcl:

I thought I posted about this on the mailing list, but when I searched I could only find I'd sent it to individual users.

In MultiMarkdown, a table of contents can be generated using the syntax:

{{TOC}}

Wouldn't it be possible to create paragraph numbering using CSS's "Counters"?

Maybe not relevant, but there is a proposed standard Scroll to Text Fragment which is only implemented in Chrome for the moment.

I would like to be able to generate a ToC on the server. My biggest concern with any webpage ToC generator, including in Fossil, is getting the header ids right so links to document sections aren't needlessly brittle. I've posted my thoughts on this problem in the paragraph id thread.

Right now I generate ToCs for my Fossil wiki pages in the browser with JavaScript in the skin's js.txt. The script (toc.js) attaches a ToC to the heading with the text "Contents". It does nothing if a heading with that text does not exist. Here is what the typical result looks like. The script is open source (MIT License) and works in IE11; feel free to use it.

Richard Hipp (drh) on 2020-09-15 14:33:17:

For writing documentation using Markdown, I'm finding that I want to have the option for the following automatically generated features:

• table of contents

• paragraph numbering

• paragraph ids.

Richard, I recall you had some code for this but decided to abandon it for potential bloat reasons or similar. I can't find those commits just now but I'm fairly sure they exist somewhere.

I suggested earlier in this thread that TH1 might be an alternative. Now I know a little more about TH1, I don't think so. And the thorny question of security keeps coming up.

Perhaps we could solve these problems by implementing the following:

1. HTML-style comment blocks within Markdown. A lot of us would find this useful (I'd use it to specify SPDX tags in documentation, for example.) So this alone would be an improvement.
2. Requiring --with-tcl to allow TOCs or any other intelligent post-processing, noting it is not the default. Anyone embedding a scripting language in their binary is accepting a degree of risk.
3. Defining a true/false value of "trusted" for Markdown documents. Untrusted documents never get to run embedded scripts. Trusted ones do. A document is trusted on the basis of DAG + user table information. Which means that a document could cease to have its scripts run in surprising circumstances, but tough.
4. For trusted documents, allow a Tcl script to be specified in the header, perhaps via TH1 calling tclEval.

Would a design like this work?

Dan Shearer

Hi Dan,

I can't find those commits just now but I'm fairly sure they exist somewhere.

This is the branch: https://fossil-scm.org/home/timeline?r=auto-toc

sean (jungleboogie) wrote on 2020-11-16 23:40:45:

This is the branch: https://fossil-scm.org/home/timeline?r=auto-toc

Thanks Sean.

To make it a little easier for people searching for the answer, here is the final word from the code in that branch on why automatic ToC will not be merged into Fossil:

Why Abandon This Change?

After putting this all together and experimenting with it, it seems like it is just extra complication. Markdown and other similar markup formats are intended to be simple. These changes are just extra complication used to insert extra boilerplate text. If you need automatic paragraph numbers and/or a TOC, perhaps you should consider using a different tool. In other words, I think this change goes against the whole philosophy of Markdown, which is to provide a simple and easy way to generate simple, readable documentation. We should not be trying to push Markdown into being a replacement for MS-Office.

This decision by drh means that ToCs are to be handled externally and provided ready-made to Fossil, in my case I will be generating Fossil Markdown with internal links. This fits with having a "make docs" step in your project and a documentation toolchain. I'm still working out what my toolchain should be.

I remember a suggestion once that Fossil ToCs be done in Javascript, but I disagree strongly with this due to accessibility and automation: both visually impaired people and search engines (etc) should be able to see a ToC if I go to the trouble of creating one.

Dan Shearer

I remember a suggestion once that Fossil ToCs be done in Javascript, but I disagree strongly with this due to accessibility and automation: both visually impaired people and search engines (etc) should be able to see a ToC if I go to the trouble of creating one.

I agree. Accessibility is very important and should be preserved. One of the many great features of Fossil is that its web interface is usable in text browsers without Javascript.

For writing documentation using Markdown, I'm finding that I want to have the option for the following automatically generated features:

• table of contents
• paragraph numbering
• paragraph ids.

That makes me think something that may be useful for others.

As a new Fossil user who currently is converting his Git repositories, I've thinking how to embed the documentation of my projects into a Fossil repository.

I always include an Asciidoctor-format <README.adoc> in my projects, which is properly rendered by GitHub. More detailed documentation is built by a Makefile from other source documents or even from the source code files, first in Asciidoctor and then in EPUB, PDF, HTML or other formats.

Asciidoctor is a light and readable markup that can represent any feature of DocBook, of course including ToC, paragraph numbering, footnotes, conditional parsing, etc. I've been using it for years for any kind of documentation, including books. Markdown is very useful in many contexts, but its limitations for complex contents make the conversion of my current projects docs undesirable, and in several cases simply impossible, not to mention the inconvenience of the different and incompatible Markdown variants. Converting my docs to the Fossil variant of Markdown is not an option in my case.

After doing some tests I have decided that the best and simplest way to integrate the current documentation into an online Fossil repository is to add an aditional target to the Makefile of the projects, to build HTML files with the Fossil headers and footers. This way I keep the advantages of writting the documentation in Asciidoctor and make it accessible into the repository.

In my case, the quick solution for my documentation needs inside Fossil, beyond what is provided by default, has been Markdeep, which provides me with most of the features I need with an acceptable JavaScript footprint (180k). You lose the particular integration of wiki pages with Fossil, like some pages that show Fossil folders (Readme.md) or those that can put snippets of the time line. I imagine that some JSON API could be offered to make them embbeddable in an HTML page.

As Fossil is mostly used by our local community to publish documentation and build micro sites, when more complex documents are needed, we can go with Pandoc, Pharo and other goodies. Our last experiment has been Brea which combines Pharo and Fossil to create a pretty versatile light solution that is kind of in between a Static Site Generator and a headless Content Management System, thanks to the way we can manage static assets, including versioning, publishing, time travel back to previous website versions, collaboration (thanks to Fossil) and also live code queries or extend data models and operations and create customized workflows (thanks to Pharo).

So, Fossil simplicity and features integration is key to used it a foundation for other projects (like web publishing or stack building), while features extension can be done relatively easy with external tools (I wish a way to script it beyond Tcl/Tk, but that's another thread ;-) ).

Cheers,

Ps: If anyone wants to see Brea in action, go to IndieWeb with Brea, where we explain and exemplify what can be done with this "hybrid" tool prototype, at this early state.

I am in a similar position, recently starting to make fossil repositories public, and including documentation generated from asciidoctor. I was delighted to find I could include the fossil header in my documents, and so have a 'single website' feel to source code + documentation.

If you like to see how this looks in one of my repositories try gemguide. Clicking on one of the sections shows page back/next links, and the fossil header remains throughout. You can see the same documentation in pdf format from the home page.

I particularly like that syntax highlighting is preserved in the embedded documentation, and probably any other feature of asciidoctor which doesn't interfere with the fossil heading style should work.

How do I build this? I have my own program to split asciidoctor's html file into 'pages' and customise it to suit. For now, I have spliced the fossil header using a bit of ruby in a rakefile, but I may add this as an option to html2pages.

I use a rakefile to control building the html, so everything is automated.

I assume by "splice the fossil header" you are inserting:

<div class='fossil-doc' data-title='Title Text'>
</div>

at the beginning/end of your html files as documented at https://fossil-scm.org/fossil/doc/trunk/www/embeddeddoc.wiki in section 1.1 'HTML Rendering With Fossil Headers And Footers'. I would have included a direct link, but there doesn't appear to be an anchor generated that I can use 8-(.

If for no other reason than getting generated link targets I would like to have header ID's autogenerated.

I like AsciiDoc too: lightweight but feature-rich.

I played with a AsciiDoctor.js and managed it to work with Fossil (version 2.11, IIRC). It took some tinkering with mimetypes, TH1, JavaScript and stylesheets in order to trigger AsciiDoc rendering on just the right /doc pages. The major drawback was the inability to autolink to Fossil-native artifacts (e.g. hexadecimal UUIDs of tickets, striked-through if closed).

I wish there were native support for AsciiDoc in Fossil. Theoretically Fossil could be extended with more VM engines, such as JS or mruby. But this is unlikely to happen. Unfortunately.

Do you have a documented way to achieve this as I've played with asciidoctor and asciidoctor.js before, even had hundreds of notes and I'd like to try to integrate it within fossil.

Right now I'm writing asciidoctor and generating HTML locally. It would be super nice to allow fossil to make use of asciidoctor.js to render .adoc files in previews, wiki etc.

The idea is I was working on a project for note taking using Elixir and Asciidoctor but I'd like to explore other venues for note taking:)

Thanks

What fossil now lacks is not a "transfer doc", Fossil now has a leading precise version/document management system(core), but being too difficult to be realized by the public. Let's say, till just now I saw Pikchr.. I know suck work has been going on, but never see a link to Pikchr. And if fossil can work without everything in one file, I believe it will soon be no less important than SQLITE. Considering currently there is no real opensource search engine (solar elastic, etc) and limited fake open source db, and git no better than fossil apart from working in a .git dir instead of a big "file".