Brew (#forgejo), as usual, is being overloaded by the scrapers.

[oxy@social.bsdlab.au]

@stefano I'm surprised more options havent appeared

[pertho@mastodon.bsd.cafe]

@stefano Have you tried blocking the IP ranges in Julian's list?

Julian Oliver (@JulianOliver@mastodon.social)

I've done the log analysis and the two biggest contributors that brought the AI crawler hits up to 2 million in a day, a 4x increase on a week prior, are ByteSpider (Singapore networks) and especially AppleBot (used for Siri and other Apple products). The parasites.txt is now >4500 lines long: https://scienceispoetry.net/files/parasites.txt

Mastodon (mastodon.social)

Or is it all coming from residential proxies?

[stefano@mastodon.bsd.cafe]

@pertho residential proxies. I blocked everything I could block, but it wasn't enough.

[oxy@social.bsdlab.au]

@pertho @stefano oooh interesting list.

I've been tinkering with ssh/httpd logs/awk and enriching the data with https://iplocate.io/ and maybe eventually greynoise and spamhaus (to get more residential proxies etc)

[tubsta@social.bsdlab.au]

@stefano Stick Bunny in with origin shield to drop the nuffs scraping your site

[stefano@mastodon.bsd.cafe]

@tubsta this would work, but I'm trying to avoid using (external) CDNs, at the moment.

[tubsta@social.bsdlab.au]

@stefano I agree with what you are trying to do as I would rather avoid CDNs but some services need it, just gotta work out the least shit ones and the ones that are Europe focused to assist here.

[tubsta@social.bsdlab.au]

@stefano FWIW I spend about $5 a month with Bunny’s CDN products for bsdlab

[mwl@io.mwl.io]

@tubsta @stefano

If you must CDN, Bunny is 100% the way to go. Contains much less suck.

I run cdn.mwl.io specifically to distribute files. It's a CDN composed of one host.

[stefano@mastodon.bsd.cafe]

@tubsta sure. Bunny is great. I have an account and use it for some services. For some time, some of the BSD Cafe contents were served by them, and it was perfect.

[tubsta@social.bsdlab.au]

@stefano They now have S3 object access for their storage nodes which has been a long time coming

[stefano@mastodon.bsd.cafe]

@tubsta oh nice! I was curious to see it implemented. I'll have a look.