If you self-host services on the internet, you may have seen waves of crawlers hammering your websites without mercy.

[bandie@yip.gay]

@Soblow Hmmm.. I think of serving a knowledge.tld just with static garbage now...

[soblow@eldritch.cafe]

Well, first documented case (on my end) of a false positive, yay...

iOS 15 + Safari, for some reason...

[jernej__s@infosec.exchange]

@Soblow Interesting, I've noticed a very similar pattern on a client's web server, which they use for hosting internal projects (which have to be available publicly) – 5000 requests per second every few hours on specific subdomains, from residential IPs, with each IP doing about 20 requests, changing the user-agent once. Nothing knowledge-like though, most of the sites run Wordpress.

[sargeros@rivals.space]

@Soblow just tried to add your article to my self hosted Readeck (a read it later service) and I think the request got caught by an anti scraper.
Guess I'll have to read the article soon to understand what happened!

[soblow@eldritch.cafe]

@Sargeros Yes, it's something expected, I have the same issue with some other tools.
Maybe it has a User-Agent I can whitelist though?

[sargeros@rivals.space]

@Soblow It seems like it announces itself as a browser https://codeberg.org/readeck/readeck/src/commit/5a979acdb2afcfe8f87d5385db778e3643322b04/internal/httpclient/client.go#L33

But don't worry, I used the browser extension to save your article and it worked fine

[soblow@eldritch.cafe]

@Sargeros pretty sure that UA got caught in the maze indeed...
good to know!

[velvet@pawb.fun]

@Soblow @NafiTheBear @terrencefoxfur

[terrencefoxfur@furryfandom.me]

@velvet @Soblow @NafiTheBear Yup they are the bane of my life. I have an ... aggressive ... filter list, and have absolutely no qualms in just blocking LLM's entirely

[soblow@eldritch.cafe]

I should do an addendum but right now, my main website is getting hammered at rates similar to what my knowledge website used to be hit at.
On the opposite, the "knowledge" website is back at "normal" background noise of <100req/min.

The banlist now contains so many IPs, and yet they still reach 6kreq/min nearly constantly.

At that point, I'm thinking about tinkering my banip tool to compute optimal subnets instead of always crafting /24 subnets.

[puffin@squawk.social]

@Soblow feel like once you have blocked the big data center it just become wack and mole since tools like those exist

Internet Sharing SDKs: a Closer Look at the Emerging App Monetization Method - Proxyway

Internet sharing SDKs are reshaping app monetization. But what do you need to know before adopting them?

Proxyway (proxyway.com)

[lilacperegrine@clockwork.monster]

@Soblow
curious, is the subnet thing using similarities in the ip to ban specific ranges?

computing optimal things sounds like a math problem and if so, im game to try it out

[soblow@eldritch.cafe]

That's frightening.
Like, legitimately, I'm scared of what that means.

I'll try to ratelimit at 10req/min per ip.

[soblow@eldritch.cafe]

@lilacperegrine Are you familiar with the concept of a quadtree?

[lilacperegrine@clockwork.monster]

@Soblow
a data structure where every node has 4 children?
im sorta familiar with it, but i havent used it much

[ifixcoinops@retro.social]

@Soblow I've been following this with interest

[soblow@eldritch.cafe]

@lilacperegrine Well, I have an intuition that the problem I'm looking to solve is akin to the construction of a quadtree:
I have a list of IPv4/32. I want to generalize this list using the properties of subnets (a n subnet contains 2 n+1 subnets).
I don't have an explaination though, and my math skills are rusty...

[eragon@pl.eragon.re]

@Soblow@eldritch.cafe @lilacperegrine@clockwork.monster Seems like a cool thing to do… if only I could use university's time to do that.

[lilacperegrine@clockwork.monster]

@Soblow
so you have a list of ips, and are using a binary(or quad) tree in order to classify them into clean vs dirty?

[soblow@eldritch.cafe]

@lilacperegrine No, not really...
I talked about quadtree because that's something I manipulated and it made met think of it.
Let's take a step back and formalize the problem:

Let's assume I have a list of unique IPv4 adresses.
They are represented on 32 bits.
I want to construct a list of subnets (so still 32 bits) that summarize the list of IPs I have.

For example, if I have 192.168.1.0 and 192.168.1.1, I could generalize this with the 192.168.1.0/31 subnet (if I'm not mistaken), which contains the previous two IPs without containing any other IPs.
If it helps, represent them in binary and find the common upper bits:

11000000.10101000.00000001.00000000 (192.168.1.0)
11000000.10101000.00000001.00000001 (192.168.1.1)

The common part is everything up to the last bit, thus the mask is a `/31` which is 255.255.255.254, or in binary `11111111.11111111.11111111.11111110`.

Now, I have tens of thousands of IPs and I'd like the smallest list of subnets that includes all bad IPs without including good IPs

I'm sure there are academic papers about this, this sounds like a problem folks must already have had