<![CDATA[Brew (#forgejo), as usual, is being overloaded by the scrapers.]]>EDIT: done, let me know if you experience problems

Brew (), as usual, is being overloaded by the scrapers.

I think I'll have to put an Anubis in front of it. I don't love those "blocks", but sometimes you need to.

]]>https://board.circlewithadot.net/topic/70493f2b-5e33-4158-9d6b-5bfb5e6846b5/brew-forgejo-as-usual-is-being-overloaded-by-the-scrapers.RSS for NodeMon, 25 May 2026 18:29:33 GMTFri, 22 May 2026 07:55:37 GMT60<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Sat, 23 May 2026 07:45:19 GMT]]>@tubsta oh nice! I was curious to see it implemented. I'll have a look.

]]>https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/stefano/statuses/116622774743578798https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/stefano/statuses/116622774743578798Sat, 23 May 2026 07:45:19 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 22:21:42 GMT]]>@stefano They now have S3 object access for their storage nodes which has been a long time coming]]>https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/c35c4f86-802e-44c6-b657-c8dd4337aeebhttps://board.circlewithadot.net/post/https://social.bsdlab.au/objects/c35c4f86-802e-44c6-b657-c8dd4337aeebFri, 22 May 2026 22:21:42 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 12:19:09 GMT]]>@tubsta sure. Bunny is great. I have an account and use it for some services. For some time, some of the BSD Cafe contents were served by them, and it was perfect.

]]>https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/stefano/statuses/116618189171364028https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/stefano/statuses/116618189171364028Fri, 22 May 2026 12:19:09 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 12:03:28 GMT]]>@tubsta @stefano

If you must CDN, Bunny is 100% the way to go. Contains much less suck.

I run cdn.mwl.io specifically to distribute files. It's a CDN composed of one host.

]]>https://board.circlewithadot.net/post/https://io.mwl.io/users/mwl/statuses/116618127497960046https://board.circlewithadot.net/post/https://io.mwl.io/users/mwl/statuses/116618127497960046Fri, 22 May 2026 12:03:28 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 11:37:55 GMT]]>@stefano FWIW I spend about $5 a month with Bunny’s CDN products for bsdlab]]>https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/4fcd6cff-6d2c-4ff1-9fa6-3eb51460f0c0https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/4fcd6cff-6d2c-4ff1-9fa6-3eb51460f0c0Fri, 22 May 2026 11:37:55 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 11:36:15 GMT]]>@stefano I agree with what you are trying to do as I would rather avoid CDNs but some services need it, just gotta work out the least shit ones and the ones that are Europe focused to assist here.]]>https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/e4f9677b-f41f-4d4c-8eab-467da3c797aahttps://board.circlewithadot.net/post/https://social.bsdlab.au/objects/e4f9677b-f41f-4d4c-8eab-467da3c797aaFri, 22 May 2026 11:36:15 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 11:24:22 GMT]]>@tubsta this would work, but I'm trying to avoid using (external) CDNs, at the moment.

]]>https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/stefano/statuses/116617973782358380https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/stefano/statuses/116617973782358380Fri, 22 May 2026 11:24:22 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 11:12:44 GMT]]>@stefano Stick Bunny in with origin shield to drop the nuffs scraping your site]]>https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/a6d69b8b-b4e8-4ab9-9cb1-9c7065ea8187https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/a6d69b8b-b4e8-4ab9-9cb1-9c7065ea8187Fri, 22 May 2026 11:12:44 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 09:12:08 GMT]]>@pertho @stefano oooh interesting list.

I've been tinkering with ssh/httpd logs/awk and enriching the data with https://iplocate.io/ and maybe eventually greynoise and spamhaus (to get more residential proxies etc)]]>https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/5c6fd338-9e65-4f2c-bff0-aeea4881f43chttps://board.circlewithadot.net/post/https://social.bsdlab.au/objects/5c6fd338-9e65-4f2c-bff0-aeea4881f43cFri, 22 May 2026 09:12:08 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 09:06:42 GMT]]>@pertho residential proxies. I blocked everything I could block, but it wasn't enough.

]]>https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/stefano/statuses/116617432411750032https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/stefano/statuses/116617432411750032Fri, 22 May 2026 09:06:42 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 09:04:44 GMT]]>@stefano Have you tried blocking the IP ranges in Julian's list?

Julian Oliver (@JulianOliver@mastodon.social)

I've done the log analysis and the two biggest contributors that brought the AI crawler hits up to 2 million in a day, a 4x increase on a week prior, are ByteSpider (Singapore networks) and especially AppleBot (used for Siri and other Apple products). The parasites.txt is now >4500 lines long: https://scienceispoetry.net/files/parasites.txt

favicon

Mastodon (mastodon.social)

Or is it all coming from residential proxies? 🤔

]]>https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/pertho/statuses/116617424690159526https://board.circlewithadot.net/post/https://mastodon.bsd.cafe/users/pertho/statuses/116617424690159526Fri, 22 May 2026 09:04:44 GMT<![CDATA[Reply to Brew (#forgejo), as usual, is being overloaded by the scrapers. on Fri, 22 May 2026 08:02:25 GMT]]>@stefano I'm surprised more options havent appeared]]>https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/aef23b52-b69b-4730-8fbc-1ceae5cb2622https://board.circlewithadot.net/post/https://social.bsdlab.au/objects/aef23b52-b69b-4730-8fbc-1ceae5cb2622Fri, 22 May 2026 08:02:25 GMT