The relevant fossil instance is sqlite.org (I'm using sqlite so I though why not test it by letting it crawl its databases documentation).

Crawling the static sqlite.org site should pose no particular issues for you vis-a-vis fossil, but its web server (althttpd) does have some small amount of anti-spider protection. Its own robots.txt is very permissive. Crawling the /src or /docsrc sites, though, both of which are fossil instances, will prove difficult for any bot, well-behaved or not.

Note that fossil does not currently support a robots.txt. It relies on the underlying web server, if any, to do host that.

depsite best efforts to prevent this, my crawler found links like ...

Unfortunately, the documentation and forum posts have double-digit-years' worth of links like that. The new redirect-to-login behavior, however, is only a few weeks old. Many such links cannot be reformulated in ways which will not trigger the login-redirect and no effort is currently being made (or planned) to weed out and rewrite those links which could be made bot-/guest-friendly.

My recommendations as the developer of a friendly crawler: ...

By and large, we don't want bots crawling fossil repositories.

To help put the "why?" of that into perspective: the canonical sqlite documentation repository is now roughly 46MB, but contains 616MB of individual artifacts, meaning that any robot which starts to crawl it will eventually suck down a bare minimum of 616MB of bandwidth, not counting links which generate dynamic content (namely diffs). For the corresponding source repo, those values are approximately 490MB and 11GB, respectively. For fossil's own repository those values are approximately 147MB and 7.1GB, respectively.

Given the number of bots scouring the network, that's an untenable load for the hoster.

A counter-recommendation: the anti-robot settings in fossil are opt-in at the repository level, meaning that you, the bot hoster, can clone the repository and then crawl that to your heart's content, using up your own bandwidth and CPU time instead of the upstream server's.

There are countless bots on the net now (IIRC, Richard recently calculated that some 80%(?) of the sqlite.org traffic is bots), and it's the many ill-behaved ones which drive the need to limit access. (Trivia: that's also the reason we use a custom-written forum software, rather than a mailing list - bot/spam traffic became uncontrollable on the mailing list.)

Crawling the static sqlite.org site should pose no particular issues for you vis-a-vis fossil.

I just need some way to tell the two apart …

Note that fossil does not currently support a robots.txt. It relies on the underlying web server, if any, to do host that.

Which makes sense if it's intended to only be part of a website.

Unfortunately, the documentation and forum posts have double-digit-years' worth of links like that.

That was exactly my point, besides that those links could have been on an external site like a blog explaining some code. (Apologies if I didn't manage to illustrate which problem lead to which recommendation).

By and large, we don't want bots crawling fossil repositories.

A counter-recommendation: the anti-robot settings in fossil are opt-in at the repository level, …

That's fine with me, if something doesn't want to be crawled I most probably don't want it in my index anyway. (I have limited server resources too)

But I'd prefer if my crawler was automatically stopped by a robots meta-tag before it runs into any spam and robot protection and an appropriate http status after. The bot not realizing that it ran into such mechanisms can cause quite a bit of traffic too and the operator realizing takes some time (in this case I didn't notice at all until testing the search index).

Another thing that would help is some documentation on how responsibly crawl a fossil instance (I didn't find one, but maybe I looked in the wrong place) even if it is as simple as a "don't crawl anything with a query parameter, and avoid path prefixes x,y and z".

Including how to automatically and reliably recognize one. (preferably from metadata, as that is easier to evaluate at the crawling state)

That would help me and others as a crawler developers and operator. Especially when running into smaller fossil instances, of which there should be quite a few out there.

To help put the "why?" of that into perspective:

That makes sense, thank you!

There are countless bots on the net now …

Wow, that's a lot, not exactly the fine way of saying »Thank you for making an awesome piece of software«.

For a completely different take on things, why not create your own copy of the sqlite docs from the sqlite docsrc repository? I've had passable results for that, stuffing the rendered results into a fossil of their own that I can then self-host without any attempt to reach out to the canonical site except for docsrc updates. The instructions for rendering the documents can be followed certainly on a low-end Linux machine with the requirements installed. I simply render to a directory under fossil management, then update the local fossil to suit.

There is one disadvantage to this approach, it works for one specific site and one specific set of circumstances. For crawling, say, the TCL site, my suggestion may not work at all.

I hope this provides a slightly different perspective.

Regards, brickviking
(Post 33)

There is one disadvantage to this approach, it works for one specific site and one specific set of circumstances …

And solves the problem for me only.

If I would be happy with the "me only" workaround solution I wouldn't have started this thread.

Also having looked at the source code: No I'm not even going to try and implement that myself, there too too many mechanisms going on and the fossil code (and quality) is no match for my limited C-skills.

Also having looked at the source code: No I'm not even going to try and implement that myself

No need: Richard added /robots.txt capability. Its utility is, however, necessarily limited to fossil instances which run as standalone servers (which is not a common configuration, and is definitely not the case for any of the sqlite.org-related projects). See the notes in that checkin for more details.

That's useful information, Thank you!

I'll probably combine that with a "if is has a query parameter, don't crawl it" policy for fossil and see how it turns out.

---

@Daniel: 🙃

fossil code (and quality) is no match for my limited C-skills.

I thought no Rust programmer fears C... 😁