Fossil Forum

How to deal with a 20,000-line forum post
Login

How to deal with a 20,000-line forum post

(1) By Richard Hipp (drh) on 2020-04-06 13:53:02 [source]

Over on the SQLite Forum, a fellow has submitted a forum post that exceeds 20,000 lines of text. (This is not abuse - I asked him to post his complete schema, which he did. He just happens to have a very big schema.)

The excessive size of this post make the forum thread difficult to read - especially on a mobile device. Is there anything we can do about this? Perhaps only show the first 50 or 100 lines of each post and include a button that says "read the rest of this post" or something?

Should we add an administrator-defined input size limit on the size of posts?

Anybody have any ideas about how best to deal with this?

(2) By Stephan Beal (stephan) on 2020-04-06 14:15:12 in reply to 1 [link] [source]

A post size limit is probably generally a good idea, but is also apparently not often needed.

We could possibly start by simply setting (in CSS) a max-height of, say, 150em, and overflow-y:scroll. That would allow the whole post but only effectively render the first 150ems of it (maybe 100 would be better?).

That would have a minimal/trivial development effort while we figure something better out, and would make that thread readable again. Most of my forum time is on a tablet, and that thread is now useless on a tablet, with no easy way to jump past the mega-post. (Not to mention that it sent a 20k-line notification to all subscribers, which may trigger a backlash from users or spam filters.)

(3) By anonymous on 2020-04-06 14:19:43 in reply to 2 [link] [source]

We could possibly start by simply setting (in CSS) a max-height of, say, 150em, and overflow-y:scroll.

Would it be a better idea to set the attributes on whole posts or the individual code blocks inside them?

(4) By Stephan Beal (stephan) on 2020-04-06 14:43:53 in reply to 3 [link] [source]

i suggest the whole post because many users don't know how to create code blocks or they post in plain text, which does not have such a thing.

(5) By Richard Hipp (drh) on 2020-04-06 15:08:41 in reply to 2 [link] [source]

I added <div class="forumPostBody">...</div> around the whole body of the forum post, and in the default CSS set a max-height of 50em together with "overflow: auto;" to cause scrollbars to appear for longer posts. The change is up for evaluation. The long post thread should be easier to read now.

Feedback requested.

(6) By Stephan Beal (stephan) on 2020-04-06 15:14:49 in reply to 5 [link] [source]

Much improved on this tablet, but it reveals a related minor problem: there's barely enough space on the right to drag/scroll the page (as opposed to the post). There's margin on the left because it's a response, but were it a top post it would be difficult to scroll down the thread with a mobile device. That said, i'm not sure that adding more right margin would generally an improvement just to support this case.

In any case, it's a usability improvement.

(7) By John (_jboy_) on 2020-04-06 18:25:44 in reply to 1 [link] [source]

On IRC the norm is: don't paste big code samples into the chat, use a pastebin-type service instead. I think it would make sense to adopt a similar norm for a forum (for instance, for code snippets longer than 100 lines), and to set an input size limit for forum posts. People trying to post longer messages could be instructed to paste their snippet elsewhere and link to it from their post.

(8) By anonymous on 2020-04-06 20:18:48 in reply to 1 [link] [source]

Aside from displaying legitimately long posts, seems in this case, using an attachment for the schema would have made more sense.

Maybe allow attachments for registered forum users?

Otherwise, encouraging use of services like pastebin, as the other poster suggested.

(9) By Richard Hipp (drh) on 2020-04-06 20:28:39 in reply to 8 [link] [source]

In the current implementation, large text/plain posts can be viewed without scrollbars by clicking on the [source] link in the title.

(10) By sean (jungleboogie) on 2020-04-06 21:18:47 in reply to 9 [link] [source]

I think that's a pretty cool implementation, though I haven't tried it on mobile browsers yet. One additional feature you may want to consider is a copy all button. Highlighting and scrolling down with the mouse will take awhile, especially with something very long like that.

Turns out it also works in the recent view as well: https://sqlite.org/forum/timeline?ss=v&y=f&n=20&vfx&nsm

(11) By anonymous on 2020-04-07 00:42:55 in reply to 7 [link] [source]

However you are depending on a third party to provide the code. So cloning the repo will not preserve all the data. This seems to be contrary 2 fossil's non mutability stance.

(12) By anonymous on 2020-04-07 02:15:14 in reply to 1 [link] [source]

Allow readers to disable this feature if they want it to always display the entire message.

(13) By Dan Barbarito (danbarbarito) on 2020-04-07 02:18:05 in reply to 1 [link] [source]

Would it be possible to instead allow forum posts to contain one or more attachments? The ticketing functionality already supports attachments (I believe), so it might make sense to extend that to forum posts as well.

(14) By sean (jungleboogie) on 2020-04-07 06:06:32 in reply to 12 [link] [source]

It can be disabled with the source view. I think that's a fair compromise.

(15) By Richard Hipp (drh) on 2020-04-07 10:49:53 in reply to 12 [link] [source]

It can be disabled using the CSS. Simply add a line like this:

    div.forumPostBody {}

If your site-specific CSS does not specify any rules for div.forumPostBody, then Fossil will automatically insert the following:

    div.forumPostBody {
      max-height: 25em;
      overflow: auto;
    }

And it is these automatically inserted rules that cause large posts to have their own private scrollbar. So if you want to prevent the large-post scrollbars, you just specify that div.forumPostBody has no CSS rules.

Maybe it would be good to provide a "CSS Cheat Sheet" in the documentation that explains how to do various minor changes, such as this.

(16) By Stephan Beal (stephan) on 2020-04-07 11:25:55 in reply to 15 [link] [source]

Maybe it would be good to provide a "CSS Cheat Sheet" in the documentation that explains how to do various minor changes, such as this.

i'll commit to that, provided nobody else is aching to. We've got a few similar hints in the forum history we can collect.

However, i believe that the poster was hoping for a way to disable that in an arbitrary forum they're visiting. That's trivial to do by toggling a CSS class on and off, the question is only where to integrate such a button.

(17) By Stephan Beal (stephan) on 2020-04-07 12:23:06 in reply to 13 [link] [source]

Possible, yes, in that the internal bits and pieces are there, but probably not something we want on a public form, especially one where anonymous users can post. Attachments become part of the permanent blockchain and can be arbitrarily large, meaning the forum's repo db can grow arbitrarily large quite quickly (consider that this forum is intended to run until at least the year 2117misref). Since attachments are one-time bundles of data with no discernible relation to other repository content, we can't reasonably apply delta compression to them (zlib compression, yes, but that's only effective for text-based posts/attachments).

That's not to say that it's unilaterally undesirable, just that it's probably not a great option for the main Fossil forum or the sqlite forum. For small/private/semi-private forums, it may well be a useful feature. (i can envisage using this forum software as a sort of blog, in which case attachments would potentially be quite useful, provided random passers-by could not upload them.)

misref = 2117 is not a project-official position, just a rough guess on my part.


  1. ^ a b Misreference

(18.1) By Stephan Beal (stephan) on 2020-04-07 12:49:21 edited from 18.0 in reply to 15 [link] [source]

Maybe it would be good to provide a "CSS Cheat Sheet" in the documentation that explains how to do various minor changes, such as this.

An initial version is at:

https://fossil-scm.org/fossil/doc/tip/www/css-tricks.md

and the permuted index has been updated.

@Richard: if you would, please read the top section about overriding default styles, to ensure that i haven't told any explicit fibs there. The behaviour, as i understand it, as different from CSS-conventional cascading, and it seems important to point that out to those who want to tweak the CSS.

@All: feel free to suggest CSS tweaks to add to that page, or improvements for the current ones.

On a related note: we should consider adding a page (a sub-page of the skin-editing process) which lists all of the default CSS selectors, as it's not necessarily intuitive exactly which selectors need to be added to a skin to override fossil's defaults. As an example which just tripped me:

The /dir page apparently uses div.columns to style that file list view, but as a skin editor my instinct would be to use the more specific div.columns.files. And that would work, but would not have quite the same behaviour, as it wouldn't(?) cause fossil to elide its div.columns rule. In this case it's six one way and half a dozen the other, as any styles in div.columns.files which match styles in div.columns will cascade/override in CSS's usual fashion, but styles in div.columns which were not explicitly overridden by div.columns.files would still be applied.

Such a page would essentially be a dump of default_css.h, and perhaps we could extend that data structure to include a brief description of what the selector is for. e.g. "forum post bodies" or "/dir page file list", pulling those comments from C-style comments in default_css.txt:

div.sidebox { /* Brief description for style-dump page. */
  float: right;
  background-color: white;
  border-width: medium;
  border-style: double;
  margin: 10px;
}

We can consider that to be my own TODO unless someone else specifically wants to take it on.

And vaguely related: 25em seems a bit low for div.forumPostBody. Many posts are now scrolling which are well within the limits of "easy readability" (such as this one).

(19) By Richard Hipp (drh) on 2020-04-07 14:33:15 in reply to 18.1 [link] [source]

Top section works for me.

(20) By ddevienne on 2020-04-14 15:04:57 in reply to 9 [link] [source]

But then there's no formatting. Maybe I'm just picky, but I'd prefer
independent ways to expand a large post (with a hint of how many lines
that would add, so I don't click it lightly when it says Expand to 20,000 more lines),
from the ability to view the raw unformatted post text.

Also, when I view Andy's post here, I'd prefer the scrollbar to be
around the SQL source portion, rather than the main post text portion.
I.e. if the number of non-code-section lines is below the limit, just
put the scrollbars around the preformatted <div>, not the whole-post.
As it stands, Andy's small last sentence at the end, below the SQL section
is easily missed otherwise.

That's how JIRA does it (which I use at work), and that reads really well.
I often have long comments in JIRA issues with lots of {noformat} sections
which all have scrollbars and are limited to N lines, but the whole comment
is never truncated.

Of course, if there are no preformatted sections, then the current behavior
can remain, but the JIRA behavior makes much more sense IMHO.

PS: Ah ah, I had to escape the <div> I wrote above! Was breaking the preview otherwise.

(21) By Stephan Beal (stephan) on 2020-04-14 23:56:59 in reply to 20 [link] [source]

Having the nested blocks be scrollable is only possible if they're recognized as blocks, which is often not the case. Many users (experience has shown) don't know how to create such blocks in markdown or fossil wiki format, and such blocks simply don't exist in plain text format. Having only the body of the post scrollable is literally the only generically-applicable option we have. Yes, it could be enhanced with JS to fully expand the post at the click of a button, but nobody's felt compelled to add such a thing yet. Such a button would need to appear on every single post, regardless of its length, because fossil has no reliable way of knowing whether any given post will be triggered into scrolling - that's a CSS-level decision.

It works in jira because they have only a single post format with wysiwyg-ish tools which allow all users to uniformly create code blocks and such. Even if fossil had those, the ability to post as plain text would effectively bypass them.

(22) By Brian Tiffin (btiffin) on 2020-04-15 06:38:42 in reply to 18.1 [link] [source]

Small typo, (I think)

...such cases require the user of !important...

user of, kinda works. :-)

(23) By Warren Young (wyoung) on 2020-04-17 18:15:16 in reply to 15 [link] [source]

div.forumPostBody { max-height: 25em;

I've been living with this change and the more recent increase to 40em, trying to get used to it, but it's not working for me. It messes up scrolling in common use cases for both desktop and mobile.

On desktop, the meaning of "scroll" with a mouse wheel or trackpad changes depending on where your cursor is on the screen, if any posts are truncated in this way.

On mobile, it's much the same, except that you'll be swiping and suddenly the screen stops moving because you've hit the end of the local post and have to find another place on the screen to swipe from in order to get it moving again. This is the more painful of the two cases: anyone thinking I'm complaining about small potatoes, try waking up with the SQLite forum on mobile sometime. :)

It's a good feature; it just needs to kick in a lot less often, so that it only happens for truly epic post lengths.

Can we try 100em or so, please?

(24) By Stephan Beal (stephan) on 2020-04-17 19:25:00 in reply to 23 [link] [source]

On mobile, it's much the same, except that you'll be swiping and suddenly the screen stops moving because you've hit the end of the local post and have to find another place on the screen to swipe from in order to get it moving again. This is the more painful of the two cases: anyone thinking I'm complaining about small potatoes, try waking up with the SQLite forum on mobile sometime. :)

As most of my online time is on a tablet, and some on my phone, i've had exactly the same pains, especially on a phone, as so many posts are taller than one screen and scrolling gets really funky with no left/right page margin to swipe.

i agree with Warren that 100-sh em is probably a better value, to keep it from kicking in until posts get "really long".

(25) By Stephan Beal (stephan) on 2020-04-27 08:39:27 in reply to 18.1 [link] [source]

Such a page would essentially be a dump of default_css.h ...

We can consider that to be my own TODO unless someone else specifically wants to take it on.

An initial implementation of this is now at:

https://fossil-scm.org/home/timeline?r=skinedit-css-list

It adds the list of built-in CSS selectors to /setup_skinedit when running in CSS-edit mode, as that page is presumably the only place where those data are potentially useful.

There are two impls: one is a UL list which shows only the CSS selectors and one is a TABLE which lists the selectors and their values. The table seems more useful but explodes the size of that page considerably, so the UL impl is currently active.

Opinions on which impl to use are welcomed. (Yes, we "could" add a URL parameter to control which is shown, but that seems like overkill.)

, and perhaps we could extend that data structure to include a brief description of what the selector is for. e.g. "forum post bodies" or "/dir page file list", pulling those comments from C-style comments in default_css.txt:

That's still a potential TODO, as the list of selectors by themselves are only marginally useful without a description of what each is for.

(26.2) By ss48 on 2020-06-13 13:54:13 edited from 26.1 in reply to 5 [link] [source]

I have spent some time reviewing this feature while reading through various forum posts.

The feature works really well on the desktop (with firefox). Scroll bars for the webpage and the post are visible and mouse scrolling behaves appropriately based on where the mouse is located. The behavior is completely predictable, and matches my preference to scroll through all content unless I use the browser's main scroll bar.

On an iPad, it's a nightmare for me. Rather than scrolling through all content as I swipe unless I go to the edge to swipe (my preference), the entire website scrolls. To scroll an individual post, I must first realize that the post is scroll-able (scroll bars are kept hidden in Safari for iOS), then tap it, and then start scrolling the post. It's not effortless and inhibits the flow of reading through the thread for me.

I would make the following recommendations:

  • It would be great if there was a way to have scrolling by swiping scroll all content in a post rather than just the posts as they are.
  • While scrolling though a post, keep the header (ex. (1) By anonymous on 2020-05-20 11:19:41 [link] [source]) and footer (reply button) persistently shown while scrolling the given post.
  • Add a tappable button/link in the footer (next to the reply button) to expand the post if possible, and a second tappable button/link to scroll to the next post (for those pesky 20000 character long posts.

I feel these changes could make scrolling on tablets more predictable for me and ensure I do not miss anything hidden in long posts filled with prose rather than code.