Let's move the relevant discussion from this thread here now:

Chris wrote:

I stopped work after converting from a single Lyx file to multiple markdown files when I realised that keeping it up to date would be very difficult with the changes that were and are happening to the Fossil code.

OK, I understand this. A larger bunch of changes will be needed in order to get the document up to date, thereafter a regular maintenance is necessary.

In addition, I concluded that the simplicity of markdown does not work well when there are lots of screenshots without ability to control where they appear in the flow of text. The existing screenshots are out of date.

Well, this can be a problem when you want to control the positioning right from the start. For HTML as output, this is not an issue as there are no page breaks. For a book-like output (PDF), we need more control, of course. I suggest to go via Pandoc to LaTeX and then do fine-tuning there if the figures do not get placed in sensible places. As Pandoc creates \begin{figure} ... \end{figure} for images, they are floating and have a good chance of being placed in nice places and we can afterwards specify some modifiers to help better placement.

Using section and paragraph numbering is not possible in plain markdown, so a CSS section heads each page. This header is not processed by Fossil for its pages.

I suggest using Pandoc's markdown again, here. It has section numbering built-in and we can also use the excellent pandoc-xnos filters to get numbered tables and figures too! They translate to our output formats HTML and PDF (via LaTeX) too. I do not expect that Fossil will render the Pandoc source correctly, but we can always provide the HTML output from Pandoc.

Pandoc, latex and pdfunite are also needed to generate the pdf version.

Why do we need pdfunite? Pandoc can read multiple markdown files and chain them together for a complete document.

Using pandoc to generate html files that stand alone without Fossil results in different output from that required for use within a Fossil repository.

Yes.

Finally, there are editorial decisions to make about the target audience and what should be included in the documentation. For instance, the chapter on TH1 is an advanced topic that does not suit the rest of the document. It's not clear if the chapter on ChiselApp should be included.

My view here is to make the book as comprehensive as possible. Exactly these advanced topics are the ones making the book interesting because it goes beyond what most people know and work with. TH1 is what gives Fossil a great deal of its flexibility and extensibility. I would love to learn more about where exactly and how I can use TH1 on Fossil to fully exploit its possibilities (also for me as a heavy user of Tcl as the parent of TH1). So, in that sense, updating the book is a fun learning experience to dive deeper into Fossil.

You are most welcome to clone the repository (available to anonymous users) and continue the work in a direction you think most suitable.

I have done so ... but I feel it would be better to get access to the FossilBook on the https://fossil-scm.org site instead of doing the whole work on yet another clone. I suggest to open a new branch on http://fossil-scm.org/schimpf-book/home and feed your approach into that, then continue from there as described above.

So much for now.

((Edited by Stephan to fix a broken forum post link.))

This was the previous forum post link: https://fossil-scm.org/forum/forumpost/40c83bba3d

I suggest you have a close look at the makefile provided at

https://www.breksta.com/library.cgi/fossilbook3/draft1/file?name=makefile&ci=tip

or in your cloned copy.

You will find extensive use made of pandoc.

Pdfunite is used because there are two separate pandoc commandline instructions to create the component files and they are assembled by pdfunite. I could not find a simple way of merging all three pandoc instructions.

Perhaps your pandoc-fu is greater than mine.

I went through three iterations in the conversion process you see. Each time started with an empty repository. The third edition has very little in common with the original Schimpf repository so I did not want to pollute it.

You have the choice to clone Schimpf first and make a branch within it. You will need commit permission to write it back to fossil-scm.org and I did not want to presume that privilege.

Thanks for the answer and the extensive work you already have done on the new edition! I looked at the makefile. I can see that you do some workarounds in order to get the title, author, disclaimer and other parts into the PDF without numbering the headers and pages and so on. This is all metadata.

I will put them into a proper metadata block (in yaml) so they will go into the right places of the html and latex template that Pandoc provides. There is no need to have them as input file to the body of the Pandoc data.

In a second step I will use the predefined template and make a custom version which puts the various metadata in the document where we want to have them (before the body).

This means, we do not need Pdfunite anymore, everything is done in one Pandoc run and the page/header numbering will be correct. We also do not need the pdf-break.md file. Again, this is formatting and nothing that Markdown should handle. It is part of the settings in the latex template.

In the end, we use markdown for content only and the templates take care of the right formatting for each output format. This is how it should be, IMHO.

I'll have a re-worked version ready some time next week.

About the repository: I still would like to see that everything is placed on fossil-scm.org so we have everything in one place. You are certainly right that the new edition has very little in common with the original book. However, the content was taken from the last revision and this is where I think a branch should start with all the new material.

To the fossil people at fossil-scm.org: is it possible to get write access to the FossilBook repository? I know I can clone it but I would like to feed back the changes. I hat the idea of having a clone on my own machine and only there. Of course, I can do backups and all that but who knows what happens ... and it is always good to let other people contribute via this central repository.

Regards, Torsten

To the fossil people at fossil-scm.org: is it possible to get write access to the FossilBook repository?

Richard is the only currently-active person capable of checking in or setting you up on that repo. i've pinged him in the developer /chat to take a look at your request. We have a waiver on file for you, so there's hypothetically nothing standing in your way.

The https://fossil-scm.org/schimpf-book/index repo is now on the same login-group as Fossil itself and there is a "stephan" user with setup privilege. Please create whatever other users are necessary.

@Torsten per Richard's post, you've now been set up in that repo (details sent per email).

Notes:

• If any others who have a license waiver on file are interested in working on the book, just let me know and we can get you set up.

• As Torsten is the only one who's shown an interest in working on the book repository in 5 years, it seems only fair that we now consider him to be "in charge" of that repo's direction and give him free reign to do so.

• /chat is active on that repo but the forum is currently not. We can, with Richard's blessing, reconsider opening up the forum if development activity explodes enough to warrant it.

Hello everyone!

I understand that this may not be of interest to everyone, but I made an attempt to start translating a new edition (on a new technology stack) into Russian. Of course, the work is far from complete, but I managed to start translating and create HTML and PDF documents for the introduction section. I did it in the empty branch 3rdEditionRussian on repository clone at my server here.

Like many, I have limited time and resources, but will time to time return to this work as part of my study of the Fossil.

I have experience with asciidoc (Python), but the experiment started suggests that pandoc looks scary at first glance, but it does its job.

Hi!

Good to see that the book will be done in various languages in the end. Did you take a look at the repository here?

https://fossil-scm.org/fossil-book/index

It reflects the current status, I just continued with it a few days ago, but my pace is still a bit too slow for my own expectations ... You will not only find the current markdown working files there but also some documentation on the production process and especially the makefile and the Pandoc templates and metadata files used in production. It's not that hard ... and Pandoc gives us a huge amount of flexibility and extensibility that we can make use of (I started with some preliminary test code for admonitions in markdown, e.g.).

The approval of one of the authors is very important to me, thank you!

Yes, I cloned from the repository at https://fossil-scm.org/fossil-book.

I want to say that it is impossible to register in the repository at this address, and anonymous users cannot even create a ticket for feedback. Where can I report bugs and provide suggestions?

Currently I copy to my branch and then add some fixes to files:

• pandoc/metadata-general.yaml
• pandoc/template-html.txt
• pandoc/template-latex.txt

If you have time, please compare the files and see my changes, in particular in the templates, by adding the "translator" property. I also needed to make an effort to support utf8 input in LaTeX.

Perhaps some changes can be taken into account in your version for all future translations. I would like to minimize edits to these templates with the next update from you.

and anonymous users cannot even create a ticket for feedback.

Practice has shown that capability to be a maintenance hassle. In this project we prefer the forum for that purpose.

Where can I report bugs and provide suggestions?

For the time being, here. If it gets to be "too much" we can activate the forum in the book's repository.

I haven't caught up on all the posts concerning the book so there's a good
chance my question has already been answered; nonetheless, what's the reason
for not updating the book in LaTeX and publishing an updated PDF? Fossil
displays PDFs nicely in the browser and hyperlinks to sections can be followed,
it's distributable and readable offline—thus a wider audience, original sources
might be reused, and perhaps the most flexible in terms of typesetting a
well-presented product. Being in LaTeX, if someone wants to use pandoc to export
to multiple other formats, they can.

ETA: I don't feel strongly enough about this to advocate further for this
direction, except to say that I'm comfortable using LaTeX that I could
contribute. Although that's likely the same for any other tool too. Also, I'm
pleased to see that some{one/ people} are keen to contribute to updating the
book!

This message on the previous forum chain provides justification for markdown as the primary source format:

https://fossil-scm.org/forum/forumpost/90f9942597

I no longer agree with that opinion thus justifying/excusing my stopping work.

I don't agree with that justification for updating the book in markdown either.
Markdown might be a suitable medium for documentation in many projects, and in
Fossil's case—we do use markdown (and Fossil wiki) for documentation. However,
this book is not the Fossil documentation—it's a book. And by using markdown
it's both limiting and more difficult than publishing a book—an already
difficult task—needs to be.

Frankly, the more I think about it—LaTeX is the best choice. But, if it had to
be anything but LaTeX, reStructuredText is a much better fit than markdown.

At the time when the short discussion about the book's future popped up on the forum, I also reckoned Markdown would not be enough, since it is really not made for having lots of figures and book-like material.

As a long-time LaTeX user I also initially thought it to be the best format. In the meantime I share the opinion that it is too talkative for non-technical (as in STEM-related) material.

My choice would have been then (would be?) AsciiDoc - it is nicer (for some definition) than rST. I even managed to mostly complete a transformation of the book into AsciiDoc.

On the other hand, Markdown has the nice benefit that Fossil groks it (well, some subset of it). Let's see how far Torsten got it, and then there will be more discussion material about what markup format and toolset we should use. If we end up with Pandoc's flavour, that would probably trigger some updates for Fossil's engine too.

if LaTeX is not an option (it probably should be, when aiming at producing a real, nicely formatted book ...), I would support the AsciiDoc proposal (especially the "AsciiDoctor" implementation): it is rather powerful. a lot more than markdown, anyway.

Yes, AsciiDoctor was that - it is anyway the most used implementation. It can produce different outputs - if there is interest, I might set up a repo (including unversioned artifacts) for browsing.

Second, third, or whatever the use of AsciiDoctor for such a task.

By the way: Asciidoc is one of the formats O'Reilly allows the authors to write real books in, for example the CouchDB book. (Asciidoctor natively transforms that to the much richer and illegible DocBook, which goes to print.)

Pdf is also natively produced, pandoc delivers the markdown for the website version.

Bottom line - it is a very readable, writable, and maintainable format, at the same time well defined and powerful.

Why use markdown for the Fossil book?

Actually, I only wanted to join an already ongoing effort to write a new edition of the FossilBook. The decision of taking markdown was taken 11 months ago and it was not taken by me. The work then started under this premise. When I joined the effort a few days ago, I accepted this premise and the work that had been done on the way. Now, I find myself confronted with the question whether that decision was the right one.

What are the criteria for a "right decision"? For me, it is not about finding the "best markup language for any job" but about choosing a markup capable of doing the job at hand and being easy to use. Markdown, in the flavour of Pandoc's markdown and with Pandoc's versatility (especially document templates and filters), is capable of doing this. There is no fancy formatting or even layout in Schimpf's book that is not possible with the tools in Pandoc. We can even do better than that and produce a book that is even more pleasing to the eye. Therefor I embraced the decision taken, accepted to contribute, and wanted to go ahead.

Chris was so kind to put quite some effort into converting the source of the book to markdown, setting up a server, hosting the repository, beginning to work on the texts quite a bit already and inviting people to join. I want to explicitly thank him for this! Also, I didn't want to hijack his work. I only felt the work had somehow stalled and found is would be a shame if no-one had the time and resources to go on with it. So I began to help. One thing I wanted to change was the place where the book will be hosted. I thought it would be better to make the development on fossil's own server in order to have it in the same central place as fossil itself, by this gain more publicity and show that this is something the community can and wants to give back to the developers of fossil.

It was not my intention to lead this effort or to steal the lead from someone else, especially not from Chris. But as he said ...

You are most welcome to clone the repository (available to anonymous users) and continue the work in a direction you think most suitable.

... I took the freedom to propose a way forward, based on his work. That is where we stand now. I am willing to kind of take the lead on this and implement my proposals (actually, I am already working on this) when Chris doesn't want to do this anymore or hasn't got the time. It was my feeling that he doesn't have strong opinions on this. Please correct me, if I am mistaken here!

Nonetheless, I can understand that you would like to have good reasons for using markdown as opposed to LaTeX or other markup such as AsciiDoc. The main reason is (as stated above): because it is possible and quite straight-forward for me. I know LaTeX well and have written a bunch of larger reports using LaTeX. However, I love the fact that markdown allows us to write once and publish in many formats (including LaTeX, PDF, HTML, and also AsciiDoc and reStructuredText and others if there will be a need for this) within one single unified framework. The way of doing this is Pandoc. Yes, markdown alone is not enough, the core markdown was not made for the purpose of writing books, but Pandoc and Pandoc's markdown are more than plain markdown and can do it. I know Pandoc and have been using it for years. And since it is able to produce the LaTeX we need for a nice book, we are not out of the LaTeX world just because we use markdown for the basic content. In the background, from within Pandoc, LaTeX and other tools are working for us to do the layout just right.

So, again, to be clear, the plan is:

• to write an updated version of the book in Pandoc's markdown
• publish it as PDF (via LaTeX)
• publish it as HTML on the website
• perhaps explore other ways to distribute the work (ebook comes to mind first)

With this plan, we have source material for the book that ...

• is simple to write, simple to read
• has a flat learning-curve for contributors
• is easy to maintain in fossil
• leaves the heavy-lifting to Pandoc

Therefor, please give me some time (a few weeks) to show you how it will work. My spare time is limited and I cannot work as fast as I would like to. Then, if you still have strong opinions against the current decision and my proposal, I will gladly hand it off to someone else.

Don't get me wrong - I did not request to switch to AsciiDoc. I was just writing about my own endeavour trying to find a suitable format. As I said, I am really looking forward to seeing what Torsten comes up with; as soon as the scaffolding is there, contributing should be easier.

And anyway: the information itself to gather and put down/together is more important and time consuming than the format. There are plenty of tools for converting, should the need arise. We need first an updated book, Fossil changed a lot and for the better in the past five (not to say eight) years since the last commit in the book repo.

I seldom have enough time to work on my own projects let alone contribute to my
favourite open source projects—so please don't feel rushed to produce anything
to justify your decisions. As Stephan rightfully explains (and as he explained
to me when I was first introduced to Warren's perfectly befitting description of
Fossil's governance), this is a do-ocracy (when, as Stephan also explained,
it's not a monarchy); so please don’t take my comments as any more than the
opining of another random user on the Internet. As previously mentioned, I’ll
likely be able to contribute irrespective of the tool used, and more than my
disagreeing to the use of markdown, I’m pleased to see you work toward updating
the book.

I still think markdown is inadequate and LaTeX best suited. I've never used
AsciiDoc, though, so can't opine on it other than if, as you say, it's nicer
than rST—it's better suited for the task than markdown so would consider it an
improvement.

But as stated in an earlier post, I don't think the toolset—irrespective of the
one chosen—will present any obstacles to my contributing. Even if it turns out to
be AsciiDoc (as it won't be difficult to grok another markup format). And more
than my objection to markdown, I'm pleased to see someone take up the task of
breathing new life into the book because it's a worthy undertaking that will add
a lot of vaulue to the project, but one that I don't have the time to undertake
myself. So to Torsten and yourself, and anyone else working on revising the
book—thank you!

I've never used AsciiDoc, though, so can't opine on it other than if, as you say, it's nicer than rST

I have the reverse opinion, so I'm curious, what are the things that in a comparison makes you prefer AsciiDoc over reST?

I am in no case an AsciiDoc expert and do not try to sell it. When I got interested in the various popular markup languages, I did my (re)search on the Web and came to gather the opinion that it is one of the most ergonomic (and rich) ML "above" Markdown. Anyone can do the same and came with its own opinion.

As for it vs. reST, "people" find the syntax more comfortable. I happen to agree. And this tends to make writing more productive. (Other nice things: it outputs natively (very nice) HTML5 and PDF, there are official implementations in Ruby, Java and JS - which among others led to browser extensions parsing it directly.)

It is a little more verbose that Markdown though, and less mainstream. So let's wait for Torsten.