The problem is that lots of crawlers do not respect robots.txt (especially those run by "AI" companies).

Thus people go for other solutions, to make it too expensive on the side of the crawler, like iocaine - https://firesphere.dev/articles/iocaine-the-deadliest-poison-known-to-ai, or anubis - https://anubis.techaro.lol

I've heard of Iocane and similar tools but never looked into them, and I guess now is the time!

i'm not familiar with your site generation code - but if you wrote it yourself, i *think* the trick would be to have it 404 when an incorrect tag has been used

How to create an error 404 page using PHP?

My file .htaccess handles all requests from /word_here to my internal endpoint /page.php?name=word_here. The PHP script then checks if the requested page is in its array of pages. If not, how can I

Stack Overflow (stackoverflow.com)

at least then the script can die() instead of yielding output. it's anyone's guess if the crawler will still continue to try generating tags when it has encountered a 404, but i *assume* they're built to avoid 404s

Ah yes, worth doing as it also improves your SEO by not having thousands of similar pages

To block the abusive subnets, I used this tool to look up the IP ranges from example IP addresses. You can see all the IP ranges for a particular host: https://www.whatismyip.com/asn/AS150436/

I then blocked using ipset/iptables but other options exist depending on your setup

Many crawlers ignore this in my experience, especially the AI ones

edit: or if you want individual tags indexed, at least reject robots for queries of more than one tag?

I'm starting to think I need to test for mutually exclusive tags, and if two or more are selected, the resulting page will have no links at all except one to go back. That will deny the bots any more links to dive into.

But maybe there are better options? I'd wager this is not a novel issue...

🧵7/7

🧵6/?

For your particular case, you should return a 404 if the URL contains both 2025 and 2026. This would stop them getting into invalid combinations. You can make it so the UI never links to these combinations by *replacing* rather than appending years if one already exists

On the resulting page, there are still 50-odd tag links available. So it loads another one, and the URL now includes "?tag=2026%2C2025". Which is nonsense, but the page still loads.

Well, there are 0 images to show on that page, but still more tags to open! So next the bot opens "?tag=2026%2C2025%2C2024"...

🧵5/?

Now: enter the web crawler bot. It finds my site. It grabs all of the links on the front page, then starts loading each one. Then it grabs all of the links on *those* pages, and starts loading all of *them*. Presumably it will stop once all links have been viewed.

🧵4/?

A human would generally make sense of the available options, and *not* select two different years simultaneously. I could even code the page so that if one year is already selected, you can't select another one.

🧵3/?

The exact thing has happened to me recently with the tags. I now require users to log in to filter by multiple tags and I've blocked the subnets of the bots

If I wanted to allow guest users to search by multiple tags, I'd probably try the following options - (1) changing it to a POST request (2) requiring JavaScript (3) using Anubis (4) looking into ip masked rate limiting, so a rate limit for like multiple ip addresses in the same block

I wrote a blog post about my situation here https://blog.rubenwardy.com/2026/04/16/contentdb-ddos/

It's here if you actually want to see it: https://bigraccoon.ca/gallery

When you filter on a tag, it adds a parameter to the URL, e.g. "domain[dot]com?tag=2026". That loads the gallery, but only displays images tagged with "2026".

You can filter further on more tags, e.g. "?tag=2026%2CPencil" ("%2C" is a URL-encoded comma), which would show images from 2026 drawn in pencil.

🧵2/?