Fossil Forum

GitHub Trip Report
Login

GitHub Trip Report

GitHub Trip Report

(1) By Richard Hipp (drh) on 2019-11-27 15:09:20 updated by 1.1

I, the founder of Fossil, was invited and attended the first meeting of the
GitHub Open Source Advisory Board in San Francisco on 2019-11-12.  This post
is a report of my adventures.

## The only skeptic in the room

GitHub was a very gracious host, providing lunch before and dinner after
the 5-hour meeting.

There were 24 people in
the room, 5 or 6 of whom were GitHub employees.  To nobody's surprise,
I was the only Git skeptic present.  But everyone was very cordial and
welcoming.  It was a productive meeting.

## It is not about Git

The GitHub employees seem to have all read at least some of my writings
concerning Git and rebase and comparing Git and Fossil and so they knew
where I stand on those issues.  They are not in 100% agreement with me, but
they also seem to understand and appreciate my point of view.  They all
seemed to agree that Git has UX issues and that it is not the optimal
Distributed Version Control System (DVCS) design.  Their argument is
that none of that matters, at least not any more.  Git is used by nearly
everyone and is the defacto standard, and so that is what they use.
Though I disagree, it is still a reasonable argument.

I offered a number of suggestions on how Git might be improved (for
example, by adding the ability to check-in revisions to comments on
prior check-ins) and their response was mostly "We don't control Git".
I replied that GitHub designed and implement the v2-http sync protocol
used by Git which as since become the only sync protocol that most
Git user will ever experience and which significantly improved the
usability of Git.  I encouraged them to continue to try to push Git
to improve.  I'm not sure how persuasive my arguments were though.

Because: It isn't really about Git anymore.  GitHub used to be a Git
repository hosting company.  But now, they are about providing other
software development and management infrastructure that augments the
version control.  They might as well change their name to Hub at this
point.

For example, the main topics of discussion at this meeting where:

   1.  Issue tracking and tickets
   2.  Code review
   3.  Automated Continuous Integration
   4.  Maintainer workflow and triage

The GitHub staff says that the four pillars of their organization are

   1.  DevOps
   2.  Security
   3.  Collaboration
   4.  Insights

You will notice that version control is not on either of those lists.
There has been a culture shift at GitHub.  They are now all about the
tooling that supports Git and not Git itself.

On the other hand, GitHub showed no hint of any desire to support
alternative version control systems like Mercurial or Fossil.  They
openly assume that Open Source (mostly) runs on Git.  It is just that
the version control is no longer their focus.  They have moved on to
providing other infrastructure in support of Open Source.

## Other Take-Aways

### Documentation Is The Best Way To Say "No"

One of the most important and also the hardest jobs of a project maintainer
is saying "no" to enhancement requests.  If you try to take on every
requested enhancement, your project will quickly loss focus and become
too unwieldy to maintain.  Participate "Paris" (whose full contact information
I was unable to obtain) says:  "Documentation is the best way to say 'no'."
In other words, it is important to document why a project does things the
way it does, as this will tend to prevent enhancement requests that cause
the project to diverge from its original intent.  I have also found that
writing about the "why" tends to help one focus on the real purpose of
the project as well.

Separately it was observed that documentation is the single most important
factor in open-source adaption.  The better your documentation, the more
likely people are to use and contribute to your project.

Implications for Fossil:

   1.  I'd like to add the ability to include 
       [PIC][PIC] and [EQN][EQN] markup in the middle
       of both Markdown and Fossil-Wiki documents.
   2.  I think this Forum feature has been very successful, but there
       are still many opportunities to improve the UX.

Addition ideas on how Fossil might be enhanced to support
better project documentation are welcomed.

### "The Tools Make The Rules"

This quote (from Tom Kremenek of the Swift project) is saying that your
workflow will be defined and constrained by the tools you use.  If you use
inferior tools your productivity will suffer.

This is the argument I've made for years about the importance of being
about to amend check-in comments and to view the descendants of check-ins.
Git partisans offer all kinds of excuses about how their workflow does not
need that.  I tend to counter with "I never needed bisect until I had the
capability."  The point is that once you have the ability to amend check-in
comments and view the descendants of a check-in, your workflow becomes more
efficient and you wonder how you ever survived without those capabilities.

What limitations
or restrictions in Fossil are limiting productivity?  Some possible ideas:

   1.   Poor tracking of files across renames
   2.   Inability to do shallow clones
   3.   Poor ticket UX
   4.   Inability to subscribe to notifications for changes to specific
        tickets or files.
   5.   Difficulty implementing actions that are triggered by new
        check-ins and/or new tickets.  This is needed for Continuous
        Integration (CI).  Enhancements to the [backoffice][backoffice]
        feature of Fossil would be welcomed.

### General agreement that rebasing is not a good thing

Even among Git partisans, there seems to be a general agreement that rebase
ought to be avoided.  The [Rebase Considered Harmful][rebase] document
is not especially controversial.

An interesting insight from the LLVM developers:  They use rebase
extensively.  But the reason is that downstream tooling controlled by
third-parties and which was developed back when LLVM was still using
SVN expects a linear sequence of changes.  The LLVM developers would like
to do more with branching, but that would break the downstream tools over
which they have no control, and so they are stuck with having to rebase
everything.  Thus rebase supports historical compatibility of legacy
tooling.  It's a reasonable argument in support of rebase, not one that
I necessarily agree with, but one that I understand.

## Summary And Concluding Thoughts

GitHub is changing.  It is important to understand that GitHub is moving
away from being a Git Repository hosting company and towards a company
that provides an entire ecosystem for software development.  Git is still
at the core, but the focus is no longer on Git.

Fossil already provides many of the features that GitHub wraps around
Git.  Prior to the meeting, someone told me that "GitHub is what make
Git usable."  Fossil has a lot of integrated capabilities that make
GitHub unnecessary.  Even so, there is always room for improvement and
Fossil should be adapting and integrating ideas from other software
development systems.

One GitHub-er asked me:  "What would it take to get SQLite
to move off of Fossil and on to Git."  Just to be clear to everyone reading
this: That will never happen.  Fossil was specifically designed to support
SQLite development and does so remarkably well.  Fossil fills a different
niche than does Git/GitHub.   Fossil will continue to
be supported and enhanced by me (as well as others) well into the
foreseeable future.

[PIC]: https://en.wikipedia.org/wiki/Pic_language
[EQN]: https://en.wikipedia.org/wiki/Eqn_(software)
[backoffice]: https://www.fossil-scm.org/fossil/doc/trunk/www/backoffice.md
[rebase]: https://www.fossil-scm.org/fossil/doc/trunk/www/rebaseharm.md

(2) By Richard Hipp (drh) on 2019-11-27 15:30:32 in reply to 1.0 [link]

Prior discussion of PIC markup in Fossil documentation was at post
[db217184de](/forumpost/db217184de).  The EQN markup would work much the same way.

Since posting that a couple of months ago, I did actually attempt to
write a PIC language interpreter in Javascript.  But I discovered that
Javascript is not a particularly good language with which to write a
parser/compiler - or at least my limited knowledge of Javascript was
insufficient for the task.  Input from readers with better Javascript
skills would be appreciated here.

I have heard that it is now possible to compile C code into WASM.  In
particular, I hear that people are compiling "sqlite3.c" into WASM and
running it in browsers.  Perhaps an alternative approach would be to
write a compact PIC translator in C (using Lemon to implement the parser)
then compile the result into WASM for download to the browser.  But I don't
understand how all of that works.  If anybody has any insights and can
help guide me through the process, that would be appreciated.

(3) By Stephan Beal (stephan) on 2019-11-27 15:41:14 in reply to 1.0 [link]

> The only skeptic in the room

But *you are not alone!*

> You will notice that version control is not on either of those lists.

That's... kind of mind-blowing. Perhaps they consider it an implicit entry in the list, since it's a level of infrastructure which underpins every aspect of what they do. A "can't see the forest for the trees" kind of thing.

> What limitations or restrictions in Fossil are limiting productivity?

My own personal entry in that list is not new: the lack of a library interface. Obviously (for those who don't know this tidbit of history) i intended [to tackle that with libfossil](https://fossil.wanderinghorse.net/r/libfossil), and was well on the way to feature-parity with the fossil binary when chronic RSI (specifically, C8 nerve inflammation/damage in both arms) effectively kicked me out of the game in late 2014 (even now, my typing capacity is still reduced to about 10-15% of what it previously was, and i am still in forced early retirement due to the years-long duration of my medical leave).

Coincidentally enough, since yesterday i've been milling over a reboot of that work, focusing initially on features which would make it easier/possible to create custom *read-only* clients, ignoring the ability to update/merge/etc. for the time being, simply to try to keep the scope in check. The motivation for that came from user elechak, who [posted a query based on the `mlink` table](/forumpost/e52eb74804). Being able to plug that type of feature one's own app/page/whatever would empower all sorts of apps we don't/can't currently have. Obviously once we have a library, the gateway is opened for any number of clients, e.g. tooling for reports, CI, and IDEs.

That said... my hands cannot reasonably commit (as it were) to that scope of project for the foreseeable future :(. Getting the library up and running again, now that fossil supports multiple hash types (libfossil predates those changes), would require an as-yet-unknown amount of effort.

*That* said, i don't think it would take too awful much to get the read-only functionality working again. So... i'll take this opportunity to bang the drum again and call for "seriously interested" coders to [get in touch](https://wanderinghorse.net/home/stephan/) and help make that happen. (Forewarning: it wouldn't be a small amount of code.) While i cannot personally commit to more than "casual" amounts of coding, my capacity for technical support is far less bounded and my capacity for moral support is practically limitless ;).

> One GitHub-er asked me: "What would it take to get SQLite to move off of Fossil and on to Git."

***LOL!*** i haven't laughed this hard since you explained the logic behind you getting taller with age!

GitHub Trip Report

(1.1) By Richard Hipp (drh) on 2019-11-27 17:44:14 edited from 1.0 updated by 1.2 [link]

I, the founder of Fossil, was invited and attended the first meeting of the
GitHub Open Source Advisory Board in San Francisco on 2019-11-12.  This post
is a report of my adventures.

## The only skeptic in the room

GitHub was a very gracious host, providing lunch before and dinner after
the 5-hour meeting.

There were 24 people in
the room, 5 or 6 of whom were GitHub employees.  To nobody's surprise,
I was the only Git skeptic present.  But everyone was very cordial and
welcoming.  It was a productive meeting.

## It is not about Git

The GitHub employees seem to have all read at least some of my writings
concerning Git and rebase and comparing Git and Fossil and so they knew
where I stand on those issues.  They are not in 100% agreement with me, but
they also seem to understand and appreciate my point of view.  They all
seemed to agree that Git has UX issues and that it is not the optimal
Distributed Version Control System (DVCS) design.  Their argument is
that none of that matters, at least not any more.  Git is used by nearly
everyone and is the defacto standard, and so that is what they use.
Though I disagree, it is still a reasonable argument.

I offered a number of suggestions on how Git might be improved (for
example, by adding the ability to check-in revisions to comments on
prior check-ins) and their response was mostly "We don't control Git".
I replied that GitHub designed and implement the v2-http sync protocol
used by Git which has since become the only sync protocol that most
Git user will ever experience and which significantly improved the
usability of Git.  I encouraged them to continue to try to push Git
to improve.  I'm not sure how persuasive my arguments were though.

Because: It isn't really about Git anymore.  GitHub used to be a Git
repository hosting company.  But now, they are about providing other
software development and management infrastructure that augments the
version control.  They might as well change their name to Hub at this
point.

For example, the main topics of discussion at this meeting where:

   1.  Issue tracking and tickets
   2.  Code review
   3.  Automated Continuous Integration
   4.  Maintainer workflow and triage

The GitHub staff says that the four pillars of their organization are

   1.  DevOps
   2.  Security
   3.  Collaboration
   4.  Insights

You will notice that version control is not on either of those lists.
There has been a culture shift at GitHub.  They are now all about the
tooling that supports Git and not Git itself.

On the other hand, GitHub showed no hint of any desire to support
alternative version control systems like Mercurial or Fossil.  They
openly assume that Open Source (mostly) runs on Git.  It is just that
the version control is no longer their focus.  They have moved on to
providing other infrastructure in support of Open Source.

## Other Take-Aways

### Documentation Is The Best Way To Say "No"

One of the most important and also the hardest jobs of a project maintainer
is saying "no" to enhancement requests.  If you try to take on every
requested enhancement, your project will quickly loss focus and become
too unwieldy to maintain.  Participant "Paris" (whose full contact information
I was unable to obtain) says:  "Documentation is the best way to say 'no'."
In other words, it is important to document why a project does things the
way it does, as this will tend to prevent enhancement requests that cause
the project to diverge from its original intent.  I have also found that
writing about the "why" tends to help one focus on the real purpose of
the project as well.

Separately it was observed that documentation is the single most important
factor in open-source adaption.  The better your documentation, the more
likely people are to use and contribute to your project.

Implications for Fossil:

   1.  I'd like to add the ability to include 
       [PIC][PIC] and [EQN][EQN] markup in the middle
       of both Markdown and Fossil-Wiki documents.
   2.  I think this Forum feature has been very successful, but there
       are still many opportunities to improve the UX.

Additional ideas on how Fossil might be enhanced to support
better project documentation are welcomed.

### "The Tools Make The Rules"

This quote (from Tom Kremenek of the Swift project) is saying that your
workflow will be defined and constrained by the tools you use.  If you use
inferior tools your productivity will suffer.

This is the argument I've made for years about the importance of being
about to amend check-in comments and to view the descendants of check-ins.
Git partisans offer all kinds of excuses about how their workflow does not
need that.  I tend to counter with "I never needed bisect until I had the
capability."  The point is that once you have the ability to amend check-in
comments and view the descendants of a check-in, your workflow becomes more
efficient and you wonder how you ever survived without those capabilities.

What limitations
or restrictions in Fossil are limiting productivity?  Some possible ideas:

   1.   Poor tracking of files across renames
   2.   Inability to do shallow clones
   3.   Poor ticket UX
   4.   Inability to subscribe to notifications for changes to specific
        tickets or files.
   5.   Difficulty implementing actions that are triggered by new
        check-ins and/or new tickets.  This is needed for Continuous
        Integration (CI).  Enhancements to the [backoffice][backoffice]
        feature of Fossil would be welcomed.

### General agreement that rebasing is not a good thing

Even among Git partisans, there seems to be a general agreement that rebase
ought to be avoided.  The [Rebase Considered Harmful][rebase] document
is not especially controversial.

An interesting insight from the LLVM developers:  They use rebase
extensively.  But the reason is that downstream tooling controlled by
third-parties and which was developed back when LLVM was still using
SVN expects a linear sequence of changes.  The LLVM developers would like
to do more with branching, but that would break the downstream tools over
which they have no control, and so they are stuck with having to rebase
everything.  Thus rebase supports historical compatibility of legacy
tooling.  It's a reasonable argument in support of rebase, not one that
I necessarily agree with, but one that I understand.

## Summary And Concluding Thoughts

GitHub is changing.  It is important to understand that GitHub is moving
away from being a Git Repository hosting company and towards a company
that provides an entire ecosystem for software development.  Git is still
at the core, but the focus is no longer on Git.

Fossil already provides many of the features that GitHub wraps around
Git.  Prior to the meeting, someone told me that "GitHub is what make
Git usable."  Fossil has a lot of integrated capabilities that make
GitHub unnecessary.  Even so, there is always room for improvement and
Fossil should be adapting and integrating ideas from other software
development systems.

One GitHub-er asked me:  "What would it take to get SQLite
to move off of Fossil and on to Git."  Just to be clear to everyone reading
this: That will never happen.  Fossil was specifically designed to support
SQLite development and does so remarkably well.  Fossil fills a different
niche than does Git/GitHub.   Fossil will continue to
be supported and enhanced by me (as well as others) well into the
foreseeable future.

[PIC]: https://en.wikipedia.org/wiki/Pic_language
[EQN]: https://en.wikipedia.org/wiki/Eqn_(software)
[backoffice]: https://www.fossil-scm.org/fossil/doc/trunk/www/backoffice.md
[rebase]: https://www.fossil-scm.org/fossil/doc/trunk/www/rebaseharm.md

(4) By Andreas Kupries (aku) on 2019-11-27 17:52:34 in reply to 2 [link]

A question about lemon. IIRC it is a LALR parser generator similar to Bison.

Is the generated parser able to execute semantic actions as the AST is built ?
This and the ability to reach back to the lexer to control its execution will be necessary for PIC.

The reason for that is PIC's macro system, which it has in lieu of proper procedures/functions. Macros can recursively call each other, and to not have this end in tears of infinite recursion this expansion has to be/is curtailed in code not executed, i.e. the untaken branches of if/else conditionals.

For the implementation of the language this means that semantic actions have to be executed as the input is lexed and parsed, and then reach back to the lexer with the outcomes of conditions to selectively skip/suppress macro expansion.

See pages 6-8 of the [dpic docs][dd].

With that in mind I wonder if it might be suitable to reuse an existing parser like `dpic`, instead of writing a new PIC parser, for compilation to WASM.

[dd]: https://ece.uwaterloo.ca/~aplevich/dpic/dpic-doc.pdf

(5) By Richard Hipp (drh) on 2019-11-27 18:05:49 in reply to 4 [link]

If you can compile dpic down into a dpic.js file that we can source and
use to render PIC on the client-side, that would be great!

(6) By Warren Young (wyoung) on 2019-11-27 18:40:00 in reply to 1.1 [link]

> Git is used by nearly everyone and is the defacto standard, and so that is what they use.

That's pretty much the "Windows" argument. Windows hasn't gone away in the face of smartphones, Chromebooks, and the cloud, but it has been pushed to the margins wherever practical. It's probably a large part of the reason that the Windows phone and tablet efforts didn't go anywhere.

It's the point of view of an entrenched victor who becomes blind to the fact that they *are* now entrenched, and consequently can no longer move.

I'm not even sure this is a conscious choice. Perhaps a better metaphor is Gulliver strapped down by thousands of Lilliputians using ropes woven from process, standardization, and interdependency. Gulliver could choose to rise, but he'd destroy all the ropes in doing so.

> \1. Poor tracking of files across renames

That's not the number one issue, just the most recent request. :)

Fixing this would be a nice quality of life improvement for those of us already using Fossil, but I doubt it factors very high in anyone's decision about whether to use Fossil, or whether to leave Fossil for something else.

> \2. Inability to do shallow clones

That goes to scalability, and it probably does have a lot to do with Fossil's adoption curve. 

It is true that most software projects aren't Linux-scale in terms of SLOC and complexity of development and thus don't benefit from elements of Git's design that serve the rare characteristics of the Linux kernel development project.

Nevertheless, it is also true that there are a lot of software projects bigger and more complex than SQLite yet still much smaller than Linux and thus could benefit if Fossil supported large-scale projects better.

> \3. Poor ticket UX

Now that we have Markdown in tickets (!!) my only serious complaint about it is the two-step filing process.

I see the value of the simplified filing interface for normal users, but when I'm logged in as a Setup user, it's annoying to not have all of the ticket fields available to me to fill out at the same time as the initial filing.

This could be controlled by a user cap saying whether that user or category of users sees the full ticket UI on filing.

Speaking of user caps, another thing I anticipate is support for various SSO technologies: LDAP, OpenID, Oauth, SAML, etc. That in turn would require some way to map groups in those systems to user categories in Fossil, which would probably have to expand beyond the current hard-coded set of four.

> \4. Inability to subscribe to notifications for changes to specific tickets or files.

That sounds like it'd be pretty easy to implement. All it wants is someone to scratch the itch.

Not me; don't care. :)

> \5. Difficulty implementing actions that are triggered by new check-ins and/or new tickets. 

Server-side Tcl hooks?

I think the hardest part is making such hooks enforcing in the face of distributed clones. If I check something into my local repo but a push to the parent repo fails because of a Tcl hook refusing the new material, what then?

> LLVM...downstream tools...stuck with having to rebase everything. 

Wouldn't a release branch (as opposed to release and version tags) solve this? Downstream projects could point at the branch they care about, which would get only a linear sequence of changes, pushed there by a release manager.

You can implement development, feature freeze, slush, and code freeze branches in this way within Fossil today, progressively restricting changes until a frozen blob of code lands on the release branch.

The only thing that might be nice in this regard is locking branches to particular users. Fossil's DVCS nature makes absolute locking impossible, but part of such organized thinking is an assumption that the users aren't actively malicious. If low-level Dev A checks something into the feature freeze branch directly, Fossil already has tools to fix that, and beyond that, it's a social/admin issue, not a tech issue.

> GitHub is moving away from being a Git Repository hosting company 

They're just moving up-stack, which is necessary to avoid becoming another commodity provider of Git service. If they'd continued to focus on Git, they'd be in direct competition with Digital Ocean and `/usr/bin/git`. They *have* to move up-stack to continue propping up those Silicon Valley salaries.

Fossil doesn't have the same need. It merely needs to remain useful to its users. I think this discussion should remain focused on that and possibly on attracting new users. But Fossil doesn't have the same business needs as GitHub, Inc., a Subsidiary of Microsoft Corporation.

(7) By Warren Young (wyoung) on 2019-11-27 18:44:20 in reply to 2 [link]

> it is now possible to compile C code into WASM

Yes, and more to the point, WebAssembly support is [approaching universal adoption][1]. It won't run in ELinks, but this is a non-core feature that those with such niche needs can ignore.

I do wonder if server-side rendering to a virtual SVG would be a better plan, though.

[1]: https://caniuse.com/#search=wasm

(8) By Warren Young (wyoung) on 2019-11-27 18:59:43 in reply to 3 [link]

> i intended to tackle that with libfossil

Let's not make the same mistake twice: `libfossil` fell into disrepair because it's a second implementation of the formats and protocols within an ecosystem that's not able to progress as fast as it wants to on a single implementation. (Thus this thread.)

Rather than resurrect `libfossil`, I'd rather see that work went into refactoring the internals of Fossil to extract a library, divorcing the CGI and CLI interfaces from a core that only deals with the repo formats.

Some of this is already done well enough that all that's needed is to draw the lines more explicitly, but I want to get to a world where the build process results in a static `libfossil2.a` that links to the Fossil binary, which can optionally be built as a shared object and linked into an app without linking to `libnss`, `libssl`, etc. because the UI is a native GUI tool that only works on local repos and thus doesn't need network clone and sync support.

The resulting library should be modular, so that if the third-party app also wants clone and sync support, it can include that as well.

Or, Plan C: expand the current web APIs to the point that one could write a full-function Fossil library that did nothing other than make HTTP API calls to a running Fossil instance.

(9) By Richard Hipp (drh) on 2019-11-27 19:15:14 in reply to 8 [link]

## Proposed Forum Enhancement

I would like to start a new Forum thread named something like "Libfossil Reboot"
based on the previous comment.  The new "Libfossil Reboot" thread would link
back to Warren's post, and Warren's post should link forward to the new thread.
But it would be a completely new thread, in as much as it is a tangent off of
the original.

In other words, I'd like to create a simple mechanism to prevent 
[thread hijacking][hijack].

Somebody with good UX sense, please suggest a suitable interface for this
new feature, or if the new idea makes no sense from a UX standpoint, please
talk me out of it.

[hijack]: https://www.urbandictionary.com/define.php?term=Thread%20Hijacking

(10) By Stephan Beal (stephan) on 2019-11-27 19:23:22 in reply to 8 updated by 10.1 [link]

Fair enough.

> Rather than resurrect libfossil, I'd rather see that work went into refactoring the internals of Fossil to extract a library, divorcing the CGI and CLI interfaces from a core that only deals with the repo formats.

That's more or less originally what the goal was. There are, however, fundamental design decisions in fossil which make "bending it" into a library unfeasible. First and foremost is its allocation failure policy: simply die. That *drastically* reduces the amount of code needed (as can be seen when comparing the same algos in fossil core to libfossil - a huge number of lines of code in libfossil are checking/handling result codes and allocation results, where fossil would simply die in most such cases). A library cannot feasibly have a die-on-OOM policy. (That is to say: i wouldn't personally use a library which i knew would outright kill my app if the library encountered an error code or couldn't allocate memory.)

That said: i would be *completely* on-board with such an effort. As Richard put it when it was proposed it in 2011, though, it would require "a Herculean effort." (Though dictionaries/style guides differ on whether *Herculean* is capitalized or not when used that way, it would certainly have a massive capital H in our context!)

> Or, Plan C: expand the current web APIs to the point that one could write a full-function Fossil library that did nothing other than make HTTP API calls to a running Fossil instance.

That's kinda/sorta what the JSON bits do, and an HTTP-based API is *essentially* a link-at-call-time DLL. However, a JSON-based API would be a particularly poor choice for certain functionality, most notably transferring binary files of arbitrary sizes, as doing so requires encoding into a form JSON can use (huge strings or massive arrays of integers (byte values), both of which are larger than the binaries themselves).

(11) By Warren Young (wyoung) on 2019-11-27 19:23:24 in reply to 9 [link]

At risk of creating a hijack of a hijack (🤷‍♂️) what if a Moderator or higher level user could mark a post as being a topic fork, which then prompts the user for a new thread title. The existing post becomes the starting post of the new thread under the new title, and unlike current first-post-in-thread, it has an "in reply to" in its header pointing back to the other thread.

For normal users, all they see is that the Reply button changes to Moved or similar. Clicking it takes them to the start of the other thread, where they can now click Reply.

Don't send the user straight to a reply: they need to see the new context first. They might then choose to reply further down-thread in the child topic.

(10.1) By Stephan Beal (stephan) on 2019-11-27 19:24:51 edited from 10.0 in reply to 8 [link]

(Edit: sorry, Richard - i didn't see your note about the threadjacking until posting this :/. Further discussion will take place in the new thread if/when it starts.)

Fair enough.

> Rather than resurrect libfossil, I'd rather see that work went into refactoring the internals of Fossil to extract a library, divorcing the CGI and CLI interfaces from a core that only deals with the repo formats.

That's more or less originally what the goal was. There are, however, fundamental design decisions in fossil which make "bending it" into a library unfeasible. First and foremost is its allocation failure policy: simply die. That *drastically* reduces the amount of code needed (as can be seen when comparing the same algos in fossil core to libfossil - a huge number of lines of code in libfossil are checking/handling result codes and allocation results, where fossil would simply die in most such cases). A library cannot feasibly have a die-on-OOM policy. (That is to say: i wouldn't personally use a library which i knew would outright kill my app if the library encountered an error code or couldn't allocate memory.)

That said: i would be *completely* on-board with such an effort. As Richard put it when it was proposed it in 2011, though, it would require "a Herculean effort." (Though dictionaries/style guides differ on whether *Herculean* is capitalized or not when used that way, it would certainly have a massive capital H in our context!)

> Or, Plan C: expand the current web APIs to the point that one could write a full-function Fossil library that did nothing other than make HTTP API calls to a running Fossil instance.

That's kinda/sorta what the JSON bits do, and an HTTP-based API is *essentially* a link-at-call-time DLL. However, a JSON-based API would be a particularly poor choice for certain functionality, most notably transferring binary files of arbitrary sizes, as doing so requires encoding into a form JSON can use (huge strings or massive arrays of integers (byte values), both of which are larger than the binaries themselves).

(12) By Warren Young (wyoung) on 2019-11-27 19:37:21 in reply to 10.0 [link]

Understand, I don't mean that your intent in creating the library was a mistake, only that we now know that it clearly didn't attract a big enough user community that you had to fend off the tendency to fork the libary when you had to give up on it, because you had multiple people all wanting to continue it. Knowing this, we shouldn't repeat the decision.

Contrast Fossil: I can see this project forking if something drastic happened to drh or to the project direction. We've got enough people actively interested in working on it that we *would* have to fight off a tendency to create multiple forks and not just one new successor project. I point that out not to show a danger to Fossil, but to show that it has a [bus factor][1] above 1.

Years now after `libfossil` stopped moving, I think we can say that it had a bus factor of exactly 1. :(

Therefore, efforts going forward should make Fossil itself more powerful, not divide precious resources.

As to the matter of difficulty, that's why I proposed my previously-unnamed Plan B as a refactoring effort, rather than reimplementation. It's something we can simply choose to strive toward, one bite at a time.

On the efficiency of a JSON API, many HTTP APIs switch content types to match the needs of the call. JSON returns may be the default, but binary and HTML returns are also legal based on context. Some calls may even have multiple forms. In that sense, Fossil UI is the HTML API to this library already, and `/raw` is the binary interface. All I'm proposing is wrapping this in a way that you can use inside, say, a Fossil plugin for a programmer's text editor.

[1]: https://en.wikipedia.org/wiki/Bus_factor

(13) By aitap on 2019-11-27 21:35:13 in reply to 2 [link]

Is there a place (someone's repo on ChiselApp?) where we could collaborate on the PIC renderer?

I seem to have created a Lemon grammar for PIC, but I cannot test it on real inputs yet, because first I'll have to hack the lexer to understand `define`s and substitute macros. (According to Kernighan and Raymond, they have to be done outside the parser, by pure sting substitution.)

Also, I haven't started the work on the "virtual machine" to interpret the AST and output SVG.

(14) By Stephan Beal (stephan) on 2019-11-27 21:45:42 in reply to 13 [link]

If you can wait about 12 hours i can set up a repo for you on fossil.wanderinghorse.net and make you the admin so you can manage collaborators. i have no idea whether the backoffice (forum) parts will work on that site, though, so no guarantees on that. If you want to go ahead and set up a repo, set up your user(s), and email me a link (see <https://wanderinghorse.net/home/stephan>) where i can download it, i can have it installed in no time tomorrow. My only requirement is that i be allowed to set myself up as an admin/superuser solely for maintenance purposes.

(15) By Stephan Beal (stephan) on 2019-11-27 22:07:17 in reply to 13 [link]

Alternately, i can set you up an empty repo. Just email me (see previous post) with your preference.

(16) By Kevin (KevinYouren) on 2019-11-29 23:17:40 in reply to 1.1 [link]

Richard,

I think "rebase" may be useful to entities who want to remove or hide errors or worse.

(17) By Saagar Jha (saagarjha) on 2019-11-30 18:11:51 in reply to 1.1 [link]

> This quote (from Tom Kremenek of the Swift project) is saying that your workflow will be defined and constrained by the tools you use.

Might you be talking about Ted, the Swift Project Lead?

(18) By Richard Hipp (drh) on 2019-11-30 18:49:20 in reply to 17 [link]

Yes, it seems likely that I wrote his name down incorrectly in my notes.
Sorry about that, Ted.

(19) By anonymous on 2019-12-01 23:45:14 in reply to 1.1 [link]

I'd like to copy this post for a personal blogpost (of course with reference) is such doing welcomed and permitted?

GitHub Trip Report

(1.2) By Richard Hipp (drh) on 2019-12-02 01:07:22 edited from 1.1 [link]

I, the founder of Fossil, was invited and attended the first meeting of the
GitHub Open Source Advisory Board in San Francisco on 2019-11-12.  This post
is a report of my adventures.

## The only skeptic in the room

GitHub was a very gracious host, providing lunch before and dinner after
the 5-hour meeting.

There were 24 people in
the room, 5 or 6 of whom were GitHub employees.  To nobody's surprise,
I was the only Git skeptic present.  But everyone was very cordial and
welcoming.  It was a productive meeting.

## It is not about Git

The GitHub employees seem to have all read at least some of my writings
concerning Git and rebase and comparing Git and Fossil and so they knew
where I stand on those issues.  They are not in 100% agreement with me, but
they also seem to understand and appreciate my point of view.  They all
seemed to agree that Git has UX issues and that it is not the optimal
Distributed Version Control System (DVCS) design.  Their argument is
that none of that matters, at least not any more.  Git is used by nearly
everyone and is the defacto standard, and so that is what they use.
Though I disagree, it is still a reasonable argument.

I offered a number of suggestions on how Git might be improved (for
example, by adding the ability to check-in revisions to comments on
prior check-ins) and their response was mostly "We don't control Git".
I replied that GitHub designed and implement the v2-http sync protocol
used by Git which has since become the only sync protocol that most
Git user will ever experience and which significantly improved the
usability of Git.  I encouraged them to continue to try to push Git
to improve.  I'm not sure how persuasive my arguments were though.

Because: It isn't really about Git anymore.  GitHub used to be a Git
repository hosting company.  But now, they are about providing other
software development and management infrastructure that augments the
version control.  They might as well change their name to Hub at this
point.

For example, the main topics of discussion at this meeting where:

   1.  Issue tracking and tickets
   2.  Code review
   3.  Automated Continuous Integration
   4.  Maintainer workflow and triage

The GitHub staff says that the four pillars of their organization are

   1.  DevOps
   2.  Security
   3.  Collaboration
   4.  Insights

You will notice that version control is not on either of those lists.
There has been a culture shift at GitHub.  They are now all about the
tooling that supports Git and not Git itself.

On the other hand, GitHub showed no hint of any desire to support
alternative version control systems like Mercurial or Fossil.  They
openly assume that Open Source (mostly) runs on Git.  It is just that
the version control is no longer their focus.  They have moved on to
providing other infrastructure in support of Open Source.

## Other Take-Aways

### Documentation Is The Best Way To Say "No"

One of the most important and also the hardest jobs of a project maintainer
is saying "no" to enhancement requests.  If you try to take on every
requested enhancement, your project will quickly loss focus and become
too unwieldy to maintain.  Participant "Paris" (whose full contact information
I was unable to obtain) says:  "Documentation is the best way to say 'no'."
In other words, it is important to document why a project does things the
way it does, as this will tend to prevent enhancement requests that cause
the project to diverge from its original intent.  I have also found that
writing about the "why" tends to help one focus on the real purpose of
the project as well.

Separately it was observed that documentation is the single most important
factor in open-source adaption.  The better your documentation, the more
likely people are to use and contribute to your project.

Implications for Fossil:

   1.  I'd like to add the ability to include 
       [PIC][PIC] and [EQN][EQN] markup in the middle
       of both Markdown and Fossil-Wiki documents.
   2.  I think this Forum feature has been very successful, but there
       are still many opportunities to improve the UX.

Additional ideas on how Fossil might be enhanced to support
better project documentation are welcomed.

### "The Tools Make The Rules"

This quote (from <s>Tom</s> Ted Kremenek of the Swift project) is saying that your
workflow will be defined and constrained by the tools you use.  If you use
inferior tools your productivity will suffer.

This is the argument I've made for years about the importance of being
about to amend check-in comments and to view the descendants of check-ins.
Git partisans offer all kinds of excuses about how their workflow does not
need that.  I tend to counter with "I never needed bisect until I had the
capability."  The point is that once you have the ability to amend check-in
comments and view the descendants of a check-in, your workflow becomes more
efficient and you wonder how you ever survived without those capabilities.

What limitations
or restrictions in Fossil are limiting productivity?  Some possible ideas:

   1.   Poor tracking of files across renames
   2.   Inability to do shallow clones
   3.   Poor ticket UX
   4.   Inability to subscribe to notifications for changes to specific
        tickets or files.
   5.   Difficulty implementing actions that are triggered by new
        check-ins and/or new tickets.  This is needed for Continuous
        Integration (CI).  Enhancements to the [backoffice][backoffice]
        feature of Fossil would be welcomed.

### General agreement that rebasing is not a good thing

Even among Git partisans, there seems to be a general agreement that rebase
ought to be avoided.  The [Rebase Considered Harmful][rebase] document
is not especially controversial.

An interesting insight from the LLVM developers:  They use rebase
extensively.  But the reason is that downstream tooling controlled by
third-parties and which was developed back when LLVM was still using
SVN expects a linear sequence of changes.  The LLVM developers would like
to do more with branching, but that would break the downstream tools over
which they have no control, and so they are stuck with having to rebase
everything.  Thus rebase supports historical compatibility of legacy
tooling.  It's a reasonable argument in support of rebase, not one that
I necessarily agree with, but one that I understand.

## Summary And Concluding Thoughts

GitHub is changing.  It is important to understand that GitHub is moving
away from being a Git Repository hosting company and towards a company
that provides an entire ecosystem for software development.  Git is still
at the core, but the focus is no longer on Git.

Fossil already provides many of the features that GitHub wraps around
Git.  Prior to the meeting, someone told me that "GitHub is what make
Git usable."  Fossil has a lot of integrated capabilities that make
GitHub unnecessary.  Even so, there is always room for improvement and
Fossil should be adapting and integrating ideas from other software
development systems.

One GitHub-er asked me:  "What would it take to get SQLite
to move off of Fossil and on to Git."  Just to be clear to everyone reading
this: That will never happen.  Fossil was specifically designed to support
SQLite development and does so remarkably well.  Fossil fills a different
niche than does Git/GitHub.   Fossil will continue to
be supported and enhanced by me (as well as others) well into the
foreseeable future.

[PIC]: https://en.wikipedia.org/wiki/Pic_language
[EQN]: https://en.wikipedia.org/wiki/Eqn_(software)
[backoffice]: https://www.fossil-scm.org/fossil/doc/trunk/www/backoffice.md
[rebase]: https://www.fossil-scm.org/fossil/doc/trunk/www/rebaseharm.md

(20) By Richard Hipp (drh) on 2019-12-02 01:07:45 in reply to 19 [link]

No objections from me.

(21) By Offray (offray) on 2019-12-02 21:41:56 in reply to 1.2 [link]

> One GitHub-er asked me: "What would it take to get SQLite to move off of Fossil and on to Git." Just to be clear to everyone reading this: That will never happen. Fossil was specifically designed to support SQLite development and does so remarkably well. Fossil fills a different niche than does Git/GitHub. Fossil will continue to be supported and enhanced by me (as well as others) well into the foreseeable future.

You could counter ask him/her: "What would it take to get full support to Fossil inside GitHub and make it a first class citizen there?" ;-)

Cheers,

(22) By anonymous on 2019-12-05 19:34:16 in reply to 16 [link]

> I think "rebase" may be useful to entities who want to remove or hide errors or worse.

If you develop on a private branch, then merge the final commit to trunk then no one need ever see any mess there might be on the branch.

(Down side: Fossil doesn't track merges from a private branch, not even in the repo where the private branch exists.)

(23) By Kevin (KevinYouren) on 2019-12-06 01:28:25 in reply to 22 [link]

Thank you, I didn't know that.

(24) By anonymous on 2020-01-27 22:04:38 in reply to 6 [link]

> If I check something into my local repo but a push to the parent repo fails because of a Tcl hook refusing the new material, what then?

What happens when you push to a parent repo where you don't have commit privilege?

(25) By Warren Young (wyetr) on 2020-01-27 22:31:29 in reply to 24 [link]

I see where you're trying to go with that point, but the difference is that when you try to push without check-in capability it's expected to be permanent until you get an account with that capability, then permanently allowed as long as your account remains active. To compare that with the denial from hooks, you're saying that a remote piece of software can effectively activate or deactivate my check-in capability on *each* artifact pushed.

Keep in mind that Fossil writes its check-ins durably first to the local blockchain, then syncs the result to the remote system. This means that once local artifacts are denied by the remote, they'll be denied until the remote changes its rules.

If we assume that the rules implemented by the Tcl commit hook are good and wisely crafted, then this means the local repo will forever have durable artifacts that can never be pushed. The local repo will keep pushing them, and the remote will keep running the commit hook logic over them, which will keep rejecting them.

All of this is fairly anti-Fossil, which tries very hard to make all clones identical.

If Fossil gets anything like this feature, I think it must be done by the hook scripts being somehow sent down with the clone and kept up to date there, so that the hook can run locally before the local blockchain gets modified.

(26) By Warren Young (wyoung) on 2020-01-28 19:15:48 in reply to 25 [link]

> I think it must be done by the hook scripts being somehow sent down with the clone

...which of course introduces portability problems: you have to ensure that the hook scripts will in fact run correctly on all of the clones, else you break those clones. "But it worked on my machine!" doesn't fly when your remote contributors stop being able to check work in just because they're on a different host OS, or are missing some third-party Tcl package, or whatever.

However, a night's sleep and a shower have provided an answer: instead of the hook script outright rejecting "bad" checkins, what if Fossil treats commit hook rejection as a signal to automatically create a branch for the new artifacts so that other work can continue on the original branch? Then the problem can be corrected in public, which is very much pro-Fossil.

We could obviously do this in a post-facto way, running such problem-detecting scripts on the server and having it automatically move artifacts, but making it part of the sync protocol with integrated hook scripts means the remote clone could be notified immediately that their check-in has been moved to a branch. The hook script could even return a detailed message explaining the problem, which could guide the remote user to a fix.

This new scheme means the content is never lost, it is committed locally and sync'd remotely, but it's set aside until the problem can be resolved.

(27) By anonymous on 2020-01-28 21:18:18 in reply to 26 [link]

> what if Fossil treats commit hook rejection as a signal to automatically create a branch for the new artifacts so that other work can continue on the original branch?

I like this idea a lot.

> but making it part of the sync protocol with integrated hook scripts means the remote clone could be notified immediately that their check-in has been moved to a branch.

I would hope that the hook script would be able to supply the new branch name, "calculated" by whatever rules the core project team deems appropriate.

Maybe the hook script could return a string with a branch name and a message separated by some reasonable delimiter.

(28) By Warren Young (wyetr) on 2020-01-28 21:51:28 in reply to 27 [link]

> the hook script would be able to supply the new branch name

Sure. It'd allow a form of namespacing, where each hook script gets its own branch naming prefix so you can tell which script shunted the commit aside. Examples:

1. **`code-style-abcd1234`**: code rejected by a site-configured automatic code style checker

2. **`ci-cd-abcd1234`**: this commit breaks the CI/CD build; implicitly an async hook

3. **`temp-code-abcd1234`**: this commit appears to contain some sort of temporary code: debug logging, <tt>FIXME</tt> comments, <tt>#if 0</tt> blocks, etc.

> Maybe the hook script could return a string with a branch name and a message separated by some reasonable delimiter.

Tcl lets you return a list of values; it isn't limited to returning a single scalar value:

```tcl
    return { 
         401,
         "code style check failed",
         "code-style-$nonce",
         $output_from_style_checker
    }
```

That is, an error code, a brief message, a branch name to move the commit to if not `null`, and an optional long message that expands on the brief message.

(29) By anonymous on 2020-12-15 16:33:25 in reply to 23 updated by 29.1 [link]

Rebase is one of two Git utilities that specializes in integrating changes from one branch onto another. The other change integration utility is git merge. Merge is always a forward moving change record. Alternatively, rebase has powerful history rewriting features. 

Rebasing is the process of moving or combining a sequence of commits to a new base commit. Rebasing is most useful and easily visualized in the context of a feature branching workflow.The primary reason for rebasing is to maintain a linear project history. For example, consider a situation where the master branch has progressed since you started working on a feature branch. You want to get the latest updates to the master branch in your feature branch, but you want to keep your branch's history clean so it appears as if you've been working off the latest master branch. This gives the later benefit of a clean merge of your feature branch back into the master branch.

 Why do we want to maintain a "clean history"? The benefits of having a clean history become tangible when performing Git operations to investigate the introduction of a regression.

In Git, there are two main ways to integrate changes from one branch into another: the merge and the rebase. In this section you’ll learn what rebasing is, how to do it, why it’s a pretty amazing tool, and in what cases you won’t want to use it.

**The Basic Rebase**

If you go back to an earlier example from Basic Merging, you can see that you diverged your work and made commits on two different branches.
The easiest way to integrate the branches, as we’ve already covered, is the merge command[.](https://www.ultimatemoldcrew.ca/bathroom-mold-removal-toronto/) It performs a three-way merge between the two latest branch snapshots (C3 and C4) and the most recent common ancestor of the two (C2), creating a new snapshot (and commit).However, there is another way: you can take the patch of the change that was introduced in C4 and reapply it on top of C3. In Git, this is called rebasing. With the rebase command, you can take all the changes that were committed on one branch and replay them on a different branch.

(29.1) Originally by anonymous with edits by Stephan Beal (stephan) on 2020-12-15 16:37:59 from 29.0 in reply to 23 [link]

Deleted

(30) By Stephan Beal (stephan) on 2020-12-15 16:43:07 in reply to 23 [link]

Reminder to moderators:

We have had one spammer today who's tried to sneak in hyperlinks to a "bathroom mold removal" product as a sneaky link on a single period, making it effectively invisible when reading the markdown-processed post but clearly visible in the unparsed content[.](like this one) Please always check the *unparsed* content of posts for new and anonymous users before approving them. This particular piece of spam was first posted by a new account (which i deleted) and then again, later in the day, as an anonymous post (which snuck past moderation but i coincidentally recognized as identical to the previous post, so deleted it).

The devious part is that such posts *appear* to be topical, but they try to sneak in spam links which are easy for the eye to overlook.