If you self-host services on the internet, you may have seen waves of crawlers hammering your websites without mercy.

[nicole4fox@datastream.cortexvoid.net]

@Soblow @StupidCamille i also use nginx mostly and i setup a fail2ban that throws 444 at crawlers

[stupidcamille@eldritch.cafe]

@nicole4fox @Soblow yeah fail2ban is a must

[peq42@mastodon.social]

@Soblow how did that work for ya after a while?
I have a website thats, sadly, not self hosted(basic php host with 4 cores) and when I tried some tar pit methods it just... overloaded its CPU until the website went down for 1h

I ended up just settling with a hidden, no-follow, no index link on every page that if you click 3x, IP bans you(405 IPs banned so far)

[soblow@eldritch.cafe]

@peq42 That's the topic of a follow-up blogpost I'm writing but, basically, I rate-limit at 1 req/m the IPs that reached a given threshold (50 req / months on my iocaine maze) with an HTTP code 429.
Behind this, if any IP hits 429 too often, they get fail2banned.
So far, I'm getting a background noise of roughly 1.5kreq/min and spikes at 5kreq/min but my server survives

[peq42@mastodon.social]

@Soblow in my case it lowered the CPU usage, but I don't think the crawlers took the hint to F off as the amount of hits/day remained the same on avg x.x

[soblow@eldritch.cafe]

@peq42 you may want to use the ipset ban method then, but keep in mind the limitations!

[niacdoial@tilde.zone]

@Soblow
okay, I finally found the time to finish reading this
(I got through most of it like 2w ago, but...)
and I have an potential idea what made the crawlers go to your main website after a while:
it's the robots.txt file itself!
my guess is that claude saw the 301 redirect and interpreted it in the same way the OG search engines interpreted links: as a way to vouch for another website's worth

[soblow@eldritch.cafe]

@niacdoial mmh...
That would explain things...
Maybe I should just put a static route on each site instead?

[niacdoial@tilde.zone]

@Soblow
maybe?
not sure it will help with the existing scrapers, since they don't seem to "abandon" a website like this
in terms of resource use, an identical static route might even be more economical than a 301.

[soblow@eldritch.cafe]

@niacdoial The reason I didn't have a static route is because of how iocaine redirection works, but I could make some careful tests.

[miranda_blue@eldritch.cafe]

@Soblow ahhh a well-thought, well-written wall of text on a subject that matters to me, with relevant illustrations and comments from cute entities. What a treat to my mind thanks for sharing!

[soblow@eldritch.cafe]

So, as planned, I published a follow-up / addendum blogpost about crawlers overwhelming my services.

It adds a few details about what happened since I put the rate limiting in place, provides an analysis of the attack sources (spoiler: big cloud providers), and some technical details about how everything works.
It also adds a few clarifications that were missing from previous post.

Read it here: https://xaselgio.net/posts/26-1.addendum-poisoning-knowledge

(I also edited the previous post to mention the follow-up)

#selfHosting #iocaine #indieWeb