The bright #LLM future, next part.
-
How can we distinguish a legitimate user who hit some URL from a scraper that distributes its operations over thousands of IP addresses?
Three ifs in a trenchcoat will get rid of the majority of those, without any additional software. The crawlers may appear complicated to defeat if you look at the user-agent only, but as soon as you look at some other headers, it turns out they're really, really, really dumb.
If you want to do more than that, and do it slightly more efficiently than a reverse proxy can, iocaine can help.
Unlike Anubis, the crawlers throwing more compute on it will not get past it, and legit visitors will (usually) remain unaware of its existence. It's in front of my own forge, happily serves ~800 req/sec (where the bottleneck is Caddy & TLS) on a €5/month potato quality VPS. It can also firewall IPs off, to further reduce load.
It does catch some "legit" crawlers like Googlebot and Bingbot, but you can allow-list those, or keep them blocked because both of those feed into LLM training too.
@algernon @mgorny cc @alderwick re the current bot attacks on our forge
"https://chronicles.mad-scientist.club/tales/surviving-the-crawlers/#three-ifs-in-a-trenchcoat"
-
The bright #LLM future, next part.
git.gentoo.org is now effectively dead, being DDoS-ed by almost a million different IPs every day. Most of them are just performing a single request at a totally random URL. How are people supposed to deal with that? How can we distinguish a legitimate user who hit some URL from a scraper that distributes its operations over thousands of IP addresses?
If you use LLM crap, you're part of the problem. You support these bastards. You should be ashamed of yourself.
@mgorny it does not help pointing to people using LLMs for legitimate reasons. It's other people using those same tools but then for nefarious purposes.
I use user-agent filtering and put Anubis in front of the Slackware git infrastructure, and that has helped immensely.
I eventually got git.gentoo.org to render and gosh! That's a lot of repositories there. Would it be an idea to distribute the cgit interface over multiple front-end servers? Like, moving all user repos to a different server? -
@mirabilos I think I get it. It's a bash-ism to redirect stdout and stderr and in (da)sh that doesn't work? @mgorny
-
@mirabilos I think I get it. It's a bash-ism to redirect stdout and stderr and in (da)sh that doesn't work? @mgorny
-
R relay@relay.publicsquare.global shared this topic
-
@davidgerard thank you!
I'm afraid on this cheap shared hosting, if I understood correctly the docs, I don't have enough permissions to run it, but I'll keep looking for stuff people could use on simple static pages...
-
@mgorny it does not help pointing to people using LLMs for legitimate reasons. It's other people using those same tools but then for nefarious purposes.
I use user-agent filtering and put Anubis in front of the Slackware git infrastructure, and that has helped immensely.
I eventually got git.gentoo.org to render and gosh! That's a lot of repositories there. Would it be an idea to distribute the cgit interface over multiple front-end servers? Like, moving all user repos to a different server?@alien@fosstodon.org, right, thank you for your concern. Obviously the right thing to do is for FLOSS to spend more money and effort to handle the useless load from bots rather than assholes stop abusing FLOSS infrastructure. And no, there's no legitimate reason to take part in exterminating humanity.
-
@mirabilos @mgorny Works pretty well for me, looking at the CPU graph.
But using LLMs to block LLMs is kinda ironic. Didn’t know they use LLMs themselves.
-
@mirabilos @mgorny Works pretty well for me, looking at the CPU graph.
But using LLMs to block LLMs is kinda ironic. Didn’t know they use LLMs themselves.
-
@mgorny oh no this forces the internet to become more and more private and intive-only instead of public
-
@algernon @mgorny cc @alderwick re the current bot attacks on our forge
"https://chronicles.mad-scientist.club/tales/surviving-the-crawlers/#three-ifs-in-a-trenchcoat"
-
@alderwick @cblgh @mgorny If I can be of any assistance, let me know, happy to help in any way I'm able to.
-
The bright #LLM future, next part.
git.gentoo.org is now effectively dead, being DDoS-ed by almost a million different IPs every day. Most of them are just performing a single request at a totally random URL. How are people supposed to deal with that? How can we distinguish a legitimate user who hit some URL from a scraper that distributes its operations over thousands of IP addresses?
If you use LLM crap, you're part of the problem. You support these bastards. You should be ashamed of yourself.
@mgorny Anything involving the web has been dead for a long time.
-
The bright #LLM future, next part.
git.gentoo.org is now effectively dead, being DDoS-ed by almost a million different IPs every day. Most of them are just performing a single request at a totally random URL. How are people supposed to deal with that? How can we distinguish a legitimate user who hit some URL from a scraper that distributes its operations over thousands of IP addresses?
If you use LLM crap, you're part of the problem. You support these bastards. You should be ashamed of yourself.
@mgorny the GHC (Haskell compiler) Gitlab is suffering similar issues.
-
@kigelia, yep. We're hosting a few huge repos (and a lot of small ones), so the load caused by crawling everything randomly (including stuff such as commit histories filtered by individual files, git blames and other stuff that's entirely redundant) prevents real people from using the service.
-
@villares no, but I went to https://iocaine.madhouse-project.org/ and faffed about a bit. I used iocaine 3 out of the box. i use nginx so i had to figure out the correct config. i added exceptions for some specific user-agents I wanted to let through.
@davidgerard @villares
I dislike iocaine because it blocks niche browsers and you cannot even contact anyone to complain (no contact data to be found on the blocking site, so the invitation „please contact the owner of the site you're visiting“ is kind of insulting) -
@davidgerard @villares
I dislike iocaine because it blocks niche browsers and you cannot even contact anyone to complain (no contact data to be found on the blocking site, so the invitation „please contact the owner of the site you're visiting“ is kind of insulting)@hierkiosk @villares unfortunately, it works real good. contactability is a question for the site being piled under with shit.
-
@hierkiosk @villares unfortunately, it works real good. contactability is a question for the site being piled under with shit.
„it works real good“ — what does this mean? Zero false-positives, zero false-negatives?
-
„it works real good“ — what does this mean? Zero false-positives, zero false-negatives?
@hierkiosk i think if you apply some thought you'll work it out
-
@mgorny iocaine 3 works against this ok
watch out for false positives
@davidgerard @mgorny I keep teetering on the edge of setting up iocaine for my own code/issues/etc self-hosting using Phorge, which keeps getting hammered into OOM-death by LLM scrapers, and I really *should*, but it just feels so darned depressing -
How can we distinguish a legitimate user who hit some URL from a scraper that distributes its operations over thousands of IP addresses?
Three ifs in a trenchcoat will get rid of the majority of those, without any additional software. The crawlers may appear complicated to defeat if you look at the user-agent only, but as soon as you look at some other headers, it turns out they're really, really, really dumb.
If you want to do more than that, and do it slightly more efficiently than a reverse proxy can, iocaine can help.
Unlike Anubis, the crawlers throwing more compute on it will not get past it, and legit visitors will (usually) remain unaware of its existence. It's in front of my own forge, happily serves ~800 req/sec (where the bottleneck is Caddy & TLS) on a €5/month potato quality VPS. It can also firewall IPs off, to further reduce load.
It does catch some "legit" crawlers like Googlebot and Bingbot, but you can allow-list those, or keep them blocked because both of those feed into LLM training too.
@algernon@come-from.mad-scientist.club At present for my own personal purposes I haven't yet set up iocaine, despite getting somewhat exasperated with outages caused by LLM scraping, due to
- I'm kinda lacklustre with proxy config stuff and there's no guide you have up for Apache2 (I'm sure I could muddle through some setup eventually though if I put actual effort into it)
- Though it's better about this not happening than Anubis, I still seem to get my main current browser of choice (Falkon) caught by Iocaine, enough so that in fact currently I cannot read the documentation as pages like just https://iocaine.madhouse-project.org/documentation/ sit loading forever even when I begrudgingly switch back to Chrome (so I assume either I've now been blocked, or your server has been hammered coincidentally). From https://chronicles.mad-scientist.club/tales/surviving-the-crawlers/#three-ifs-in-a-trenchcoat I got for example
x-request-id: 0ia4BzigBrtETmjH0S2ndso, as per the text there at the bottom after the AI-aimed gobbleygook, I am telling you
but I'm working on the poisoning side! 
