I have an obnoxious problem with crawlers eating bandwidth on my personal web site—not just the fact that crawlers consume so much bandwidth, but rather a behaviour that is absolutely next-level.

[jsstaedtler@mastodon.art]

I'm not fundamentally opposed to web crawlers, I would actually love it if my work is more discoverable. But this is such an obnoxious situation that I'm forced to accomodate or protect against.

I'm starting to think I need to test for mutually exclusive tags, and if two or more are selected, the resulting page will have no links at all except one to go back. That will deny the bots any more links to dive into.

But maybe there are better options? I'd wager this is not a novel issue...

🧵7/7

[redstrate@mastoart.social]

@jsstaedtler a dumb solution would be to tell robots to not index the page (robots meta tag) if there is any tag queries, which i assume you can do via PHP.

edit: or if you want individual tags indexed, at least reject robots for queries of more than one tag?

[rubenwardy@hachyderm.io]

@redstrate @jsstaedtler

Many crawlers ignore this in my experience, especially the AI ones

[jsstaedtler@mastodon.art]

@vga256 I'm sharing a paid host with a friend. Thanks to relatively low combined popularity, we can get away with a cheap plan, but I really don't want random bots to ruin that

[rubenwardy@hachyderm.io]

@jsstaedtler

To block the abusive subnets, I used this tool to look up the IP ranges from example IP addresses. You can see all the IP ranges for a particular host: https://www.whatismyip.com/asn/AS150436/

I then blocked using ipset/iptables but other options exist depending on your setup

[redstrate@mastoart.social]

@rubenwardy @jsstaedtler it would at least help with the legitimate ones!

[rubenwardy@hachyderm.io]

@redstrate @jsstaedtler

Ah yes, worth doing as it also improves your SEO by not having thousands of similar pages

[jsstaedtler@mastodon.art]

@redstrate Ah, this sounds promising! I don't want to make my site invisible on the greater Web by blocking all bot crawlers, but I'd be fine with them only loading URLs with no queries/parameters (anything after a ?). I'll look into that meta tag, though I acknowledge the other reply here that bots can happily ignore that.

[foobarsoft@mastodon.social]

@jsstaedtler This, I think, is why so many people have moved to having Cloudflare in front of their sites. To block/limit badly behaved bots.

[vga256@mastodon.tomodori.net]

@jsstaedtler ah okay. i imagine that probably limits you from any making any apache/nginx configuration settings changes (e.g. IP blocklists)

i'm not familiar with your site generation code - but if you wrote it yourself, i *think* the trick would be to have it 404 when an incorrect tag has been used

https://stackoverflow.com/questions/1381123/how-to-create-an-error-404-page-using-php

at least then the script can die() instead of yielding output. it's anyone's guess if the crawler will still continue to try generating tags when it has encountered a 404, but i *assume* they're built to avoid 404s

[jsstaedtler@mastodon.art]

@cb I also use the .htaccess method to "block" specific agents, so they simply get thousands of 0 byte responses. Whenever it's a known LLM/AI scraper, I'm happy with that solution (and IP blocking ones that don't present a unique user agent).

I've heard of Iocane and similar tools but never looked into them, and I guess now is the time!

[gemelen@mammut.moe]

@jsstaedtler @redstrate

The problem is that lots of crawlers do not respect robots.txt (especially those run by "AI" companies).

Thus people go for other solutions, to make it too expensive on the side of the crawler, like iocaine - https://firesphere.dev/articles/iocaine-the-deadliest-poison-known-to-ai, or anubis - https://anubis.techaro.lol