If you self-host services on the internet, you may have seen waves of crawlers hammering your websites without mercy.

[soblow@eldritch.cafe]

@flann Okay, I read it, it's highly interesting and it would've helped if I saw this earlier

There are things from your blogpost I'll likely try later too

[pandro@fedi.imowl.net]

@Soblow@eldritch.cafe
A nice read! Nothing too new for me as I was following you live on that journey but good to hear you found something that helps!

Also using iocaine for my services I at one point thought "why not let iocaine also serve garbage to empty user-agents?". That'd also catch a lot of the vuln scanners that are convinced I'm using wordpress (I'm not).

There's a surprising number of legitimate traffic you wouldn't expect to not set a UA ​​

[soblow@eldritch.cafe]

@pandro That could be something, but for example the "fursona lookup" tool didn't have a User Agent set (until I told the author)...

[seldenotter@blimps.xyz]

@Soblow "I spent the last few years building up a tolerance to iocaine powder."

[icewolf@masto.brightfur.net]

@Soblow Hah... I think we're getting iocaine or something when trying to read the article on our phone (iOS Safari, on iOS 15 or something like that). Haven't tried our desktop. Pretty meta, though.

[bandie@yip.gay]

@Soblow Hmmm.. I think of serving a knowledge.tld just with static garbage now...

[soblow@eldritch.cafe]

Well, first documented case (on my end) of a false positive, yay...

iOS 15 + Safari, for some reason...

[jernej__s@infosec.exchange]

@Soblow Interesting, I've noticed a very similar pattern on a client's web server, which they use for hosting internal projects (which have to be available publicly) – 5000 requests per second every few hours on specific subdomains, from residential IPs, with each IP doing about 20 requests, changing the user-agent once. Nothing knowledge-like though, most of the sites run Wordpress.

[sargeros@rivals.space]

@Soblow just tried to add your article to my self hosted Readeck (a read it later service) and I think the request got caught by an anti scraper.
Guess I'll have to read the article soon to understand what happened!

[soblow@eldritch.cafe]

@Sargeros Yes, it's something expected, I have the same issue with some other tools.
Maybe it has a User-Agent I can whitelist though?

[sargeros@rivals.space]

@Soblow It seems like it announces itself as a browser https://codeberg.org/readeck/readeck/src/commit/5a979acdb2afcfe8f87d5385db778e3643322b04/internal/httpclient/client.go#L33

But don't worry, I used the browser extension to save your article and it worked fine

[soblow@eldritch.cafe]

@Sargeros pretty sure that UA got caught in the maze indeed...
good to know!

[velvet@pawb.fun]

@Soblow @NafiTheBear @terrencefoxfur

[terrencefoxfur@furryfandom.me]

@velvet @Soblow @NafiTheBear Yup they are the bane of my life. I have an ... aggressive ... filter list, and have absolutely no qualms in just blocking LLM's entirely

[soblow@eldritch.cafe]

I should do an addendum but right now, my main website is getting hammered at rates similar to what my knowledge website used to be hit at.
On the opposite, the "knowledge" website is back at "normal" background noise of <100req/min.

The banlist now contains so many IPs, and yet they still reach 6kreq/min nearly constantly.

At that point, I'm thinking about tinkering my banip tool to compute optimal subnets instead of always crafting /24 subnets.

[puffin@squawk.social]

@Soblow feel like once you have blocked the big data center it just become wack and mole since tools like those exist

Internet Sharing SDKs: a Closer Look at the Emerging App Monetization Method - Proxyway

Internet sharing SDKs are reshaping app monetization. But what do you need to know before adopting them?

Proxyway (proxyway.com)

[lilacperegrine@clockwork.monster]

@Soblow
curious, is the subnet thing using similarities in the ip to ban specific ranges?

computing optimal things sounds like a math problem and if so, im game to try it out

[soblow@eldritch.cafe]

That's frightening.
Like, legitimately, I'm scared of what that means.

I'll try to ratelimit at 10req/min per ip.

[soblow@eldritch.cafe]

@lilacperegrine Are you familiar with the concept of a quadtree?

[lilacperegrine@clockwork.monster]

@Soblow
a data structure where every node has 4 children?
im sorta familiar with it, but i havent used it much