@Soblow in my case it lowered the CPU usage, but I don't think the crawlers took the hint to F off as the amount of hits/day remained the same on avg x.x
P
peq42@mastodon.social
@peq42@mastodon.social
Posts
-
If you self-host services on the internet, you may have seen waves of crawlers hammering your websites without mercy. -
If you self-host services on the internet, you may have seen waves of crawlers hammering your websites without mercy.@Soblow how did that work for ya after a while?
I have a website thats, sadly, not self hosted(basic php host with 4 cores) and when I tried some tar pit methods it just... overloaded its CPU until the website went down for 1hI ended up just settling with a hidden, no-follow, no index link on every page that if you click 3x, IP bans you(405 IPs banned so far)